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□.Introduction 



0.1. Kernel representations. Let G be a classical real group and let 
K be its maximal compact subgroup. Consider the Riemannian noncompact 
symmetric space G/K . There exists a hermitian symmetric space 

G/k D G/K 

such that ^ 

dimR G/K ^ - dimg G/K 

and G/K is a totally real submanifold in G/K (the list of enibeddings G/K 
G/K see in Section 6). We say that the symmetric space G/K D G/K is the 
hermitization of the symmetric space G/K. 

We define a kernel representation p of the group G as a restriction of an 
unitary highest weight representation p of G to the subgroup G. By well- 
known Harish-Chandra construction, highest weight representations of G are 
natural representations in spaces of (scalar- valued or vector- valued) holomorphic 
functions on G/K. We say that p is a scalar valued kernel representation if p is 
realized in scalar- valued holomorphic functions on G/K. 

The kernel representations are deformations of L'^(G/K) in some precise 
sense explained in Subsection 1.13. 

The purpose of this paperQ is to obtain the Plancherel formula for scalar- 
valued kernel representations (see formula (2.6)-(2.15)). 

There were different reasons for interest, which was attracted by kernel rep- 
resentations in last 5 years (see |, §, §, ||-[|6|, [Q, [^-^, H, 0), 
and we will formulate reasons which are the most closed to the author. In first 
place, there are many explicit analytical formulas related to kernel representa- 
tions (I hope that this paper also confirms this statement). Secondly, spectra of 
kernel representations are very rich^. Thirdly, the kernel representations have 

1 supported by grants RFBR-98-01-Q0303 and RFBR 96-01-96249 

^This work is continuation of worka,^5|, but logically it is independent on these papers. 
Our main result was announced in pm . 

^ The most interesting spectral problems of noncommutative harmonic analysis which were 
intensively investigated in last 20 years are 
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some interaction with function theorj^, see Q, Q, js^. Forth, the kernel 
representations also are closely related to Olshanskii constructions of represen- 
tations of infinite dimensional groups U(p, cxd), 0(p, cxd), Sp(p, cxd) (see p^). 

0.2. Bibliographical comments. Let G/K be itself an hermitian sym- 
metric space (Lc. G = \]{p,q), Sp(2n,M), S0*(2n), S0(n,2)). Then its her- 
mitization G/KisG/Ky-G/K. A kernel representation of G in this case is a 
tensor product of a highest weight representation pa of G and a lowest weight 
representation of G. In short paper Q published in 1978 Berezin announced 
nice Plancherel formula for the sufficiently large parameter a = (3 of highest 
weight (see below Subsections 1.10-1.11). In this case^ the kernel representation 
is equivalent to the representation of G in l^iGjK). Berezin died soon after 
this and the proof never was published^. Berezin work didn't attract serious 
interest in this time (see only papers |]l5|,||46|] on related subjects). 

Secondly, the kernel representations appeared in G.I. Olshanskii and my work 
which was partially announced in , |^ and partially published in ||3^ . This 
work cocerned in vector valued kernel representations of the groups G = 0(p, g), 
U(p, g), Sp(p, g) for small values of the highest weight. The main topic of our 
work was investigation of discrete part of spectra of the kernel representations 
and construction of "exotic" unitary representations of G by simple functional 
theoretical tools. 

In the middle of 90-s interest to kernel representations increases (we list 
some publications: [||, §, ||, ||, 0, @,|§). In 1994 Upmeier and 
Unterberger published proof of Berezin formula (see also [^)Q Van Dijk and 
Hille obtained the complete Plancherel formula for rank 1 groups. Olaffson 
and Orsted proved that for a large highest weight a scalar valued kernel 
representation of G is equivalent to the representation of G in L'^{G/K). 

In paper there was defined B-function for arbitrary classical noncom- 
pact Riemannian symmetric space. For the symmetric cones GL(n,R)/0(n), 
GL(n, C)/U(n), GL(ri, H)/Sp(n) these B-integrals coincides with Gindikin B- 
function constructed in ||ll|(1964) (see also exposition in Q). The construction 

— on pseudoriemannian symmetric spaces 

— Howe dual pairs (and the problem of decomposition of on Stiefel manifolds which are 
in some sense equivalent to Howe dual pairs) 

Each representation, which occurs in spectra of Howe dual pairs, occurs in spectra of some 
kernel representation. Converse statement is false. The a priori explanation of this phe- 
nomenon is contained in l33|. 

I think that spectra of kernel representations and spectra of on pseudoriemannian sym- 
metric spaces essentially differs. A priori embedding of spectra of TplMlp. q. K)J\ ] (r K) x 
U(p — r,q,K.)) for K = R, C,H tQ_apectra of kernel representations is discussed in Pq], |33| . 

* For instance, in the work |38[ we use functional theoretical arguments for construction 
of singular unitary representations of groups U{p,g,K) 

^Tensor products for Sho (M.) = U(l, 1) were earlier investigated by Pukanszky P5[ and by 
Vershik, Gelfand and Graev |53| , see also p?! 

® 20 years later I heard some reminiscences about this proof but I can not reconstruct proof 
itself. It essentially differs from Unterberger— Upmeier proof and my proof . 
Their result also covers groups Ee, Ef. 
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of Q for special cases of parameters gives sorae integrals of Siegel Q , Hua Loo 
Keng 1^ , Unterberger-Upmeier , Arazy-Zhang |Q . The Plancherel formula 
for scalar-valued kernel representations of all classical groups for large values of 
parameter is easily reduced to these B-integrals. In this case a kernel represen- 
tation is equivalent to the representation of G in L^{G/K) and the spectrum of 
the kernel representation is supported by the principal nondegenerate unitary 
series. 

The case of small values of highest weight was discussed in paper . In 
this case the spectrum of kernel representation is quite intricate and work 
contains natural decomposition of a kernel representation on subrepresentations 
having relatively simple spectra. 

The purpose of the present parer is to obtain the complete Plancherel formula 
for the kernel representations in the scalar-valued case. 

0.3. Contents. Main part (Sections 1-5) of the paper deals with the series 
G = 0(p,g)|. 

Section 1 of the paper contains preliminaries. We discuss the definition of the 
kernel representations and simple a priori properties of the Plancherel formula. 
We also formulate some necessary properties of spherical functions. 

Basic results are formulated in Section 2. For large values of a (where a is 
the parameter of a highest weight) the Plancherel measure has the form 

E{o^) n + (P + '?)/2 + Sfc)l'^(5)ds 

where fH(s) is the Gindikin-Karpelevich density (see(1.42)-(1.43)), 

si,...,SpeiR (0.1) 

(this notation means that Re sj — 0) and E(a) is a meromorphic factor. 

Assume q — phe sufficiently large. Let us move the parameter a from -|-oo to 
0. After passing across the point a — ■^{p + q) — 1 there appears an additional 
piece of the support of the Plancherel measure. This piece is defined by the 
conditions 

si ^ a~ ^{p + q) + 1; S2,...,SpeiM (0.2) 

After passing across the point a — ^{p + q) — 2 the third piece of the support 
of the Plancherel measure appears: 

si = Q - i(p + q) + 1, S2 = a - i(p + q) + 2; S3, . . . ,Sp e iR (0.3) 

®This case is the most complicated and all difficulties existing for other series exist also for 
0{p,q). For all other series our proof is more simple. 
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After passing across the point a = ^{p + q) — 3 we obtain two additional com- 
ponents of the support 

Si = a— + S2 — a—^{p+q)+2, S3 = a— i(p+q)+3; S4, . . . , Sp e iR 
and 

si = a - + + 3; S2,...,SpGiM. (0.4) 

etc. After passing across the point a — {q — p)/2 we obtain the first one-point 
piece 

si = a - ^{p + q) + I; S2 = a - ^{p + q) + 2, . . . , Sp = a - ^{p + q) + p 

This means that our representation has a subrepresentation entering discretely. 

At the point a — p — 1 the component (0.1) of the support disappears. At 
the point a ^ p — 2 components (0.2), (0.4) also disappear, etc. 

Theorems 2.2-2.4 contain the complete description of this process and give 
the Plancherel density on each component of the support. Interpretation of 
these pieces is given in [|6|, in the present paper this is not discussed. 

The nature of spectra of kernel representations is explained in [36|P|. 

For an integer negative a our construction gives the Plancherel formula for 
some finite dimensional representation of 0(p, q) (see Subsection 2.6 of the pa- 
per). 

Section 3 is based on |3^ and contains evaluation of the B-integral (see 
formula (3.2)-(3.4)). For instance, in the case p = q our B-integral is given by 



n 



det[(i? + i?*)/2]^^~^^+^ 

' ■detiR + R')-PdR= (0.5) 



B.+m>afJi dct[l + i?]"^'""^+^ 



const 



r(Afc - (p + k)/2 + l)r(gfc -Xk-{p- k)/2) 
T{ak-p + k) 



where the integration is given over the space of dissipative p x p real matrices 
R and the symbol [A]j denotes the left upper j x j block of a matrix A. 

B-Integral allows to obtain the Plancherel formula for a > ^{p + q) — 1. In 
Section 4 we construct the analytic continuation of the Plancherel formula to 
arbitrary a. 

In Section 5 we prove positive definiteness of spherical functions which ap- 
pears in the right side of the Plancherel formula. 

^The discrete part of spectra of kernel representations consists of singular unitary repre- 
sentations having quite interesting properties. For instance, these irifinite dimensional (non 
highest weight) representations have Gelfand-Tsetlin bases, see [bsl; the problems of 

decompoailion of restrictions and tensor products for these representations also seems rich, 
see |BC| , ]38| ) . In |43{ , |pq] it was shown, that these representations admit inductive limits as 
5 — > 00. Certain representations of this type appear in spectra discussed in |50|, [|25| 
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Section 6 contains a discussion of other series of classical groups. The B- 
integrals for other series or real classical groups are evaluated in and a 
generalization of the consideration of Subsections 1,4, 5 to other series is quite 
trivial. Hence, we give only short remarks and also give the Plancherel formula in 
the form which slightly differs from Theorem 2.2. Author intentionally considers 
the series 0{p, q) (and not so-called 'general case') to do the exposition more or 
less self-closed. I try to avoid formal logical dependence on recent papers and 
also minimize using machinery of representation theory of semisimple groups as 
far as it is possible^; I also try to avoid notations demanding long explanations. 

Acknowledgements. I am very grateful to G.I.Olshanskii, V.F.Molchanov, 
and B.Orsted for numerous discussions of the subject. I thanks H.SchlichtkruU, 
G. van Dijk, and A.Dvorsky for discussions, comments and references. 

1. Preliminaries. 

A. Positive definite kernels. 

The subject of the paper is the analysis in a family of hilbert spaces defined 
by positive definite kernels. The notion of positive definite kernel and associated 
machinery are quite old (see ||5l|, Q, j2^) but not widely known. In this 
Section we briefly discuss elementary properties of the positive definite kernels 
and associated hilbert spaces. 

1.1. Positive definite kernels. Let iJ be a hilbert space with a scalar 
product < •,• >, let A" be a subset in H. Consider the function L{x,y) on 
X X X defined by 

L{x,y) ^< x,y > 
Obviously for all xi, . . . , Xn S A we have 

(L{xi,Xl) ■■■ L{xi,Xn)\ 
\ ^0 (1.1) 
L(Xn,Xi) ■■■ L(Xn,Xn)J 

Let A be an abstract set. A function L{x,y) on A x A is called a positive 
definite kernel if it satisfies the conditions 

1. L{x,y) = L{y,x) 

2. For any xi, X2, . . . , x„ G A inequality (1.1) holds 

Let L{x,y) be a positive definite kernel on X. Then where exists a hilbert 
space H = H[K] and a system of vectors Vx & H enumerated by points x € X 
such that 

^"we needs in some basic jiroperties of spherical functions, all necessary information is 
contained in Helgason book |17| , chapter 4 
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1. < Vx,Vy >H^ L{x,y) 

2. the linear span of the vectors Vx is dense in H . 
The family Vx is called a supercomplete basi^f^. 

This construction is natural in the following sense. Let H' be another hilbert 
space and let v'^ be another system of vectors satisfying the same conditions. 
Then there exists the unique unitary operator U : H ^ H' such that Uvx — v'x 
for all X G X. 

If X is a separable metric space and the kernel L{x,y) is continuous, then 
the hilbert space H[L] is separable. 

In Subsections 1.2-1.3 and 1.4 we discuss two ways of "materialization" of 
the space H[L\. 

1.2. Scalar product in the space of complex-valued measures. As- 
sume X be a separable complete metric space. Let be a complex-valued 
measure (charge) on X with a compact support. Consider a vector 

v{^.) = / Vxdfi{x) G H[L] 
Jx 

Thus, we obtain a way to represent elements of H[L\. Obviously 



<v{^),v(v) >H[L\^ I L{x,y)d^i{x)dv{y) 
Jx-kx 

Let us say the same construction more formally. Consider the linear space 
JA{X) of all compactly supported complex-valued measures on X . Consider the 
scalar product in JA[X) defined by the formula 



<H,v>=l L{x,y)d^i{x)du{y) 

J XxX 

We obtain a structure of a prehilbert space in JA{X) and the space H[L\ is 
the hilbert space associated with the prehilbert space JA{X) (elements of the 
supercomplete basis corresponds to measures supported at points). 

1.3. Scalar products in spaces of distributions. Assume AT be a 
smooth manifold and the kernel L{x,y) be smooth. Denote by B the space 
of compactly supported distributions on X. Consider the scalar product in 2) 
given by the formula 



<X,V'>= {Kix,y),x{x)(E>(p{y)} (1.2) 

where brackets {■, ■} denote the pairing of smooth functions and distributions. 
Consider the hilbert space H associated with the prehilbert space D. Denote 
^-distribution supported at a point x hy 6x- Obviously 

< Sx,Sy >= L{x,y) 



Other terms for Vx are overfilled basis or system of coherent states. 
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Hence, we can identify H with H[L\ and the vectors 6x with elements of the 
supercomplete basis Vx- 

Remark 1. The space of distributions equipped with scalar product (1.2) 
is not complete. This means that some vectors of H[L] can not be represented 
by distributions. 

Remark 2. Scalar product (1.2) in ID can be degenerated. This means that 

a vector h G H can be represented by a distribution in various ways. 

1.4. The embedding of H[L] to the space of functions on X. For 

arbitrary h £ H[L] we consider the function 

.fh{x) :=< h,Vx >H[L] 

on the space X. Obviously, the map /i is an embedding of H[L] to the 

space of functions on X. We denote the image of the embedding by H°[L]. By 
construction, the space H° [L] has structure of a hilbert space. 

Lemma 1.1. Assume X be a separable m,etric space and the kernel L{x,y) 
be continuous. Let hj converges to h. Then fhj converges to fh uniformly on 
compacts. 

Proof. Let y c X be a compact set. Let x gY. Then 



\fh,{x) - fh{x)\ = I < {hj - h,Vx >H[L] I < 

^ \\hj - h\\ ■ \\vx\\ = \\hj - h\\ ■ ^L{x,x) < \\hj - h\\ ■ . /maxL(x,a;) Kl 

V x&Y 

Obviously, the function (pa{x) G H°[L] associated with the vector Va € H[L] 
is given by the formula 

<fa{x) = L{x,a) (1.3) 

Lemma 1.2. Let X be a locally compact metric space and the kernel L{x,y) 
be continuous. Then Junctions fh € H°[L] are continuous. 

Proof. The linear span of the functions (pa is dense in H°[L]. Then we 
apply Lemma 1.1. IE 

Lemma 1.3. (Reproducing property) For any f G H°[L], x G X the 
following identity holds 

f{x)=< f,ipx>Ho[L] (1.4) 

Proof. Let / = fh. Then 

< fh,V>x >H°[L] = < h,Vx >H[L]= fh{x) ^ 

Remark. Equation (1.4) gives a nonexplicit description of the scalar prod- 
uct in 11° [L] and this description is sufficient for many purposes. Another way 
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of description is the following identity. Let e„(a;) e H°[L] be an orthonormal 

basis. Then 

Hx,y) = '^en{x)e„{y) 

(proof: consider < (px,en >) 

Lemma 1.4. Let fl be an open domain in C" and let L{z,u) he a pos- 
itive definite kernel on il. Let L{z,u) he holomorphic in the variahle u and 
anti-holomorphic in the variahle z. Then all elements of the space H°[L\ are 
holomorphic functions on Q,. 

Proof. It is a corollary of Lemma 1.1. IE 

1.5. Operations with positive definite kernels. 

Lemma 1.5. a) Let Li{x,y), L2{x,y) he positive definite kernels onX. Then 
Li(x, y) + L2{x, y) is a positive definite kernel 

b) Let Li{x,y), L2{x,y) he positive definite kernels on X. Then the kernel 
Li{x,y)L2{x,y) is positive definite 

c) Let Lj{x,y) he positive definite kernels and Lj(x,y) converges to L{x,y) 
point-wise. Then L{x, y) is a positive definite kernel. 

d) Let Km{x,y) be a family of positive definite kernels enumerated hy points 
of some measure space M with positive measure fi. Assume that the integral 

K*{x,y)= / Km{x,y)dn{m) 

JM 

converges for all x,y £ X. Then K*{x,y) is positive definite. 

e) Let L{x,y) he a positive definite kernel and let X{x) he a function on X. 
Then the kernel M{x,y) = X{x)X{y)L{x,y) is positive definite. 

Proof, a) Let Vx (resp. Wx) be the supercomplete basis in H[Li] (respec- 
tively H[L2]). We consider the system of vectors Vx (B Wx & H[Li] (B H[L2]. 
Then 

Li{x, y) + L2{x, y) =< Vx ® Wx,Vy ® Wy > 

b) Proof is similar, H[LiL2] C H[Li] ® H[L2] 

d) This is consequence of a) and c). 

e) Indeed, 7J[M] = H[L] and the supercomplete basis in H[M] consists of 
vectors j{x)vx where Vx is the supercomplete basis in H[L]. 

1.6. Positive definite kernels on homogeneous spaces. Let F be a 

group acting on X and let a positive definite kernel L{x,y) be F-invariant 

L{gx, gy) = L{x, y) for all g G F, .r, y X 

Obviously, for each g € T where exists the unique unitary operator U{g) : 
H[L] H[L] such that 

Uvx = v„x for all x & X 
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Then 



U{gi92) = U{gi)U{g2) 



Hence, U (g) is an unitary representation of F. 

Let X be a r-homogeneous space, X — T/K, let xq be a Jf-fixed point. 

Let L{x,y) be a F-invariant function. Then L is completely defined by the 
function 

l{y) L{xQ,y) 

Indeed, let u,z £ X. Then u = gxo for some element g in F and 

L{u, z) = L{gxo, z) = L{xo,g^^z) = l{g^^z) 

Moreover, for any j (z K we have 

l{y) = L{xo,y) = L{'jXQ,jy) = L{xQ,jy) = ^(7^) 

We see that the function is a invariant function on T/K. 

We also can consider a iiT-invariant function l{y) as function on double cosets 
K\T/K. 

We see that there is the canonical correspondence between 3 following sets: 

- T -invariant Junctions on T/K x T/K 

- K -invariant Junctions onT/K 

- Junctions on K \ T/K . 

We say that a A'-invariant function on T/K or a function on K \ T / K is 
positive definite if the associated kernel on T/K x T/K is positive definite. 

1.7. On ii'-invariant vectors in representations of F. Let F, K, xq be 

the same as above. Let p be an unitary representation of F in a hilbert space 
H . Assume that there exists a if-invariant vector v E H and assume w be a 
cyclic vecto^^ 

Consider the map T/K H given by the formula 

gxQ p{g)v 

(the image of the map is the F-orbit of the vector v). Then the function 
L{giXa,g2Xo) :=< p{gi)v, p{g2)v >h 

is a F-invariant positive definite kernel on T/K. 

Hence, we can identify the hilbert space H with the space H[L]; the super- 
complete basis in H consists of vectors p{g)v. 

The function l{y) in our case is the matrix element < p{g)v,v > and our 
construction (Segal-Gelfand-Naimark construction) reconstructs the represen- 
tation p by its matrix element. 

^^This means that the hnear span of vectors p{g)v is dense in H. 
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B. Kernel representations. 



Assume p ^ q 

1.8. Pseudoorthogonal group 0(p, g). Consider the linear space ® C 
equipped with the indefinite hermitian form 

p g 

J{{x,y),{u,v))=J2xjUj~^ykVk; {x,y),{u,v)eCP®C'' (1.5) 

The pseudounitary group U(p, q) is the group of all linear operators g = ^ ^ 

in ® C preserving the form J(-,-)- In other words, a matrix g G V{p,q) 
satisfies the condition 



a (3\ fl \ fa f3\ _ fl 
7 6) [O -I U 5 - lo -1 



(1.6) 



The pseudoorthogonal group 0{p, q) is the subgroup of U(p, q) consisting of real 
matrices. Below in Sections 1-5 by the symbol G we denote the group 

G = 0{p,q) 

By K we denote the subgroup 0{p) x 0(g) C G consisting of matrices having 
the form . It is a maximal compact subgroup in G. 

1.9. Matrix balls. By Bp^q(C) we denote space of all complex pxg matrices 

z having norm < 1 (where a norm is the norm of the operator v vz from the 
euclidean space C^' to the euclidean space C; remind that is the maximal 
eigenvalue oi z*z). 

By Bp^q(R) wc denote the space of real p x q matrices with norm < 1. 

The group U(p, q) acts on the matrix ball Bp^g(C) by fractional linear trans- 
formations 



z 



[g] 



{a + z^)-^{P + z5) (1.7) 



This action is transitive and the stabilizer of the point ^ = is the subgroup 
U(p) X U(g). Hence, Bp^g(C) is the symmetric space 

Bp,g(C)=U(p,9)/U(p)xU(g) 

In the same way, Bp^g(M) is the symmetric space 

Bj,,,(M) = G/K = 0{p,q)/0{p) x 0(9) 

Arbitrary symmetric space admits unique up to factor invariant measure. For 
the space Bp^g(IR) the 0(p, g')-invariant measure is given by the formula 

dX{z) = det(l - ^*2)-(P+«)/2rf^(2) (1.8) 
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where diJL{z) is the Lebesgue measure on Bp^g(M). 

1.10. Berezin kernels. Theorem 1.6[^ . The kernel 

La{z, u) = det(l — z*u)^°' 

on the matrix hall Bp.g(C) is positive definite if and only if 

a = 0, 1,2, . . 1 or a>p-l (1.9) 

Thus, for a satisfymg the Berezm condition (1.9), we obtain the hilbert 
spaces Ha :— H[La] and H° :— H°[La]. The function La{z, u) is anti-holomorphic 
in z and hence by Lemma 1.4 the space H° consists of holomorphic functions 
on the matrix ball Bp.q(C). 

Remark. For a > p + q — 1 the scalar product in H° can be represented in 
the form 

<f,g >„= C{a) I f{z)^Aei{l ~ z* zr~P~Uii{z) 

where dfi{z) is the Lebesgue measure on Bp ,j(C) and C{a) is the mcromorphic 
factor defined by the condition < 1,1 >a— 1- In particular H°^q is the Bergman 
space. For a — q we obtain the Hardy space . The scalar product in this 
case is given by the formula 



< fiz),g{z)>q^ f{z)g{z)dv{z) 

Jz*z = l 

where di'{z) is the unique U(p) x U((7)-invariant measure on the setf^ zz* = 1. 
For other values of parameters there exist integral formulas including partial 
derivatives but they are not simple (see |l[]). 

Remark. For a > p—1 the space Ha contains all polynomials on Bp g. For 
a = 0, 1, . . . , p — 1 all functions / € Ha satisfy some system of partial differential 
equations. For a = our space contains only constants. 

Proposition 1.7. (|||l, Q) a)For any 5 = ( " ^ ) e V(j),q) the operator 



Ta{g){z)f{z) = f{{a + zc)-\b + zd)) dct(a + zc)-" (1.10) 

is unitary in H" 

Remark. If a is not integer, then 

det(a + zc)"" = det a"" det(l + zca^^)"" = 

= |deta|-" •e""(''^''S'^°"'+2'^'=^Met(l + zca-i)-" (1.11) 

^'^see Berezin |Q (1975), see also Gindikin jll) , Rossi, Vergne , Wallach [ ^^ , see also a 
recent exposition in |b|. 
^*Schtiefel manifold 
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is a multi-valued function. It is easy to show that ||ca~-^|| < 1. Hence, (1 + 
zca~^)~" is a well-defined single- valued function on the matrix ball Bp,g(C). 
Hence, expression (1.11) has countable family of holomorphic branches on Bp^g(C) 
and formula (1.10) defines a countable family of well-defined operators which 
differs by constant factors e^'^'^"'. 

Proof. Consider the supercomplete basis <fix{z) = det(l — x*z)~" in H°. 
A calculation show that 

Ta{9~^)Vx{z) = det(a + zcj^ip^ig] (z) 

The simple identity 

det(l - a;[»l(t/[sl)*) = det(l - xy*) det(a + a;c)-^ det(a -|- yc)-^ 

implies 

Obviously 

Ta{gi)fa{g2) = e^~f^{gig2), where meZ 

If a is integer, then Ta is a linear representation of U(p, q). If a is not integer, 
then Ta is a projective representation of U(p, q) or a linear representation of the 
universal covering group \J{p,q)"^ of the group \J{p.q). 

1.11. Kernel representations of 0{p, q). Kernel representation Ta of the 
group G = 0{p,q) is the restriction of the representation Ta to the subgroup 
0(p, q). We also say that the function f{z) = 1 is the marked vector in H°. We 
denote this vector by S. 

Remark. A kernel representation is a linear representation. Indeed, we can 
wright I deta|~"(l -|- zca~^)~°' instead of (l-H). 
Lemma 1.8. The vector is 0{p,q)- cyclic. 

Proof. Let Q C Ha be a subspace containing the G-orbit of S. This 
orbit consists of functions (1.11) and hence the functions det(l + zca~^)~°' are 
contained in Q. But the point ca~^ is the image of under the fractional linear 
transformation (1.7). Since the action of 0(p, g) is transitive on Bp^g(C), the 
subspace Q contains all functions Vu = det(l -|- zu*)~°' where u G Bp^g(M). 
Furthermore, since the family Vu depends on u holomorphically, Vu €: Q for all 
u G Bp^g(C). But Vu is the supercomplete basis in H°. Hence, Q = H°. Kl 

Lemma 1.9. Any 0{p,q) -invariant subspace in H° contains an 0{p,q)- 
invariant vector. 

Proof. Assume ff° = R(BQ where R, Q are invariant subspaces. Assume 
that R hasn't an 0{p) x 0(q')-invariant vector. Then the projection of !B to i? 
is zero, and hence E G Q. But S is cyclic. Thus, Q = H^. lEl 
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1.12. Another description of the kernel-representations. Let a sat- 
isfies Berezin conditions (1.9). By Lemma 1.5.e) the kernel 

, , det(l-zz*)"/2det(l-Mw*)"/2 
det(l — zu*)"- 

on Bp,g(M) is positive definite. A simple calculation show that the kernel 

is 0(|5, (7)-invariant. Hence, we obtain an unitary representation of the group 
0{p,q) in the hilbert space H[Ma] — H°[Mc,] (see Subsection 1.6). The group 
0{p,q) acts in H°[Ma] by substitutions 

f{z)^ f{{a + zc)-\b + zd)) (1.13) 

The marked vector S in this model is the element of the supercomplete basis 
corresponding to the point G Bp,g(R). 

Let us define the canonical unitary 0(p, g)-intertwining operator 

A:H°[L„]^H°[M^] 

Let / e H°[L„], let z e Bp,q(IR). Then 

Af (z) = f{z) det{l - zz*)""/^ 

This map transforms elements of the supercomplete basis in H° [La] to ele- 
ments of the supercomplete basis in H° [M„] . 

1.13. Limit as a — !■ oo. Let A be the 0{p, g)-invariant measure on Bp_g(IR) 
(see (1.8)). Denote by Co the space of continuous functions on Bp_g(IR) with 
a compact support. If e Cq, then ip{z)X{z) is a complex valued measure on 
Bp q. Hence (see Subsection 1.2), we obtain the scalar product in the space Cq 
given by 

<ip,ip>=Aa Mociz, u)ip{z)ip{u) dX{z)dX{u) (1.14) 

JBp.,(R)xBp.,(R) 

Let us define the normalization constant Ac, by the condition 
Aa=(f {l-zz*)°'dz)~^ 

^JBp,,(R) ^ 

(it is a Hua Loo Keng integral, see (3.5)). Obviously Ma{z, z) = 1 and Ma{z, u) < 
1 if z ^ u. It is easy to see that the sequence Ao,Ma{z,u) approximates the 
distribution 5{z — u). Thus, the limit of scalar products (1.14) as a — > oo is 



<(/?,V'>= / ip{z)'(p{z) dX{z) 
In this sense the limit of kernel representations as a ^ oo is the space L'^{G/K). 
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We emphasis that the action of 0(p, q) in L^(0(p, q)/0{p) xO{q)) and in all 
spaces H°[Ma] is given by the same formula (1.13) and only scalar product in 
the space of functions varies. We will see that the spectrum of the representation 
Ta and the structure of Plancherel formula essentially depends on a. 

1.14. Preliminary remarks on the Plancherel formula. Our pur- 
pose is to obtain a decomposition of the kernel representation Ta on irreducible 
representations. 

An irreducible representation of G = 0{p, q) is called spherical if it contains 
a JC- fixed vector. This vector is called spherical vector. Remind that the space 
of if-fixed vectors for G has dimension or 1 (Gelfand theorem, see for instance 
[ pT| , Theorem 4.3.1 and Lemma 4.3.6). Denote the set of all unitary spherical 
representations of 0{p,q) by Gsph 

Remark. The explicit description of this set is not known. Parametrization 
of all (generally speaking nonunitary) spherical representations of 0{p,q) is 
simple and it is given below in Subsection 1.17). 

By Hp we denote the space of a spherical representation p, by £_{p) we denote 
the spherical vector in Hp whose length is 1. 

Lemma 1.10. Decomposition of the kernel-representation Ta has the form 

Ta{g) - / „ P{9)dva{g) (1.15) 

where Va is a Borel measure on Gsph- 

Remark. For the definition of direct integrals of representations and the 
abstract Plancherel formula see, for instance, |2^, 8.4. 

Proof. By Lemma 1.9, the decomposition contains only spherical repre- 
sentations. Hence, by the abstract Plancherel theorem the representation Ta{g) 
has the form 

3 = 1 

where 

Rj^ I p{9)d<{g) 

and the measure ly^'^^ is absolutely continuous with respect to ly^ for all j. The 
number >c can be 1,2, . . . , oo. We must prove that >c = I. 

All K-fixed vectors in Rj are functions having the form ipj{p)^{p) where 
^Pj (p) is a z^;^-measurable function on Ggph ■ 

Consider the projection S'^'^'^-' of the marked vector S to i?i © i?2. Since the 
vector S is cyclic in whole space, its projection must be cyclic in i?i © i?2. The 
vector S^^^'^) has the form 

{MpMp),MpMp)) eRi®R2 
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Obviously the cyclic span of ^(^'^^ in Ri R2 contains only vectors 
satisfying the condition 

V2{p)qi{p) = ^i{p)q2{p) 

If i/^ ^ we obtain a contradiction, since the cyclic span of S'^'^^^^ is a proper 
subspace in i?i i?2 ■ ^ 

1.15. Normalization of the Plancherel measure. The measure Va in 
(1.15) is defined up to equivalence of measures]^. 

The image of the marked vector S in the direct integral (1.15) is some func- 
tion (p{p)^{p) where ^{p) is an unit JC-fixed vector in Hp. It is convenient to 
assume 

ifip) = 1 (1.16) 

This assumption uniquely defines the measure Va- 

Remark. Assumption (1.16) is not restrictive. Indeed, let us assume that 
the image of S in (1.15) is a function "f{p)C{p)- Then the Plancherel measure is 
completely defined by this assumption and it equals to 



After normalization (1.16) we obtain the following equality of matrix ele- 
ments 



p{g)dva{p) 

Gsph 



1,1 > (1.17) 



or 



< r„(g)s,s >H„- / < p{g)ap),ap) >H, dv^ip) (1.18) 

Gsph 

Conversely, assume that we know a measure Va on Gsph satisfying condition 
(1.18). Then it satisfies condition (1.17). Hence, the representations Tq and 
/g h P^9^'^^°' have the same matrix elements, and therefore they are canonically 
equivalent (see Subsection 1.7). 

The marked vector 5 is _fC- invariant, therefore (see Subsection 1.6) we can 
consider the matrix element 

S„(g) :=<T„(.9)S,S> 

as a function on G/K or a function on K \ G/K. Vectors S and Ta{g)'E. are 
elements of the supercomplete basis in H[Ma\, therefore the function can 
be easily evaluated. 

-'^measures fi, v are equivalent if there exists a function x such that X 7^ almost everywhere 
(in sense of v) and ^ = xv. 
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In the matrix ball model of G / K the function !Bq, is given by the formula 

^^(z) =det(l ze^p,q (1.19) 

Let us obtain the formula for !Ba as function on if \ G/K. Denote by a* the 
element of 0(p, q) given by the matrix 



/ cosh t\ 



sinh ti 



at 



cosh 



sinh t„ 



sinh ti 



cosh 



sinh t„ 



cosh f„ 















1 



V 



\ 






1 / 



(1.20) 

It is easy to show that arbitrary element g oiG = 0{p, q) can be represented in 
the form 

g = kiatk2] where fci, k2 £ K 

The collection of parameters t= {ti, . . . ,tp) is uniquely defined up to permuta- 
tions of tj and reflections 

{h, ...,tp)^ {oiti,. . .,aptp) (1.21) 

where aj = ±1. 

We denote by A the subgroup in 0{p,q) consisting of all elements at- We 
denote by Dp the group of transformations of MP generated by permutations of 
coordinates and reflections (1.21). 

We identify the set K\G/K with the set of Dp-orbits on A. 

In coordinates {t\,. . . , tp) the matrix element Sq, is given by the formula 



'Ba{tl,...,tp) = cosh-" tk 



(1.22) 



fe=i 



Hence, we must obtain the expansion (1.18) of the function Sq, given by formula 
(1.19) or (1.22) in positive definite spherical functions. 

Our purpose in Section C is to give an expression for spherical functions. 



C. Spherical representations and spherical transform 



1.16. Parabolic subgroup. Consider the space M^'®K' equipped with the 
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indefinite symmetric form J defined by formula (1.5). A subspace y C © R'' 
is called isotropic if the form J is zero on V. 

An isotropic flag V in ® is a family of isotropic subspaces 

V : Fi C V2 C • • • C Fp; where 6im.Vj = j 

The flag manifold 5" is the space of all isotropic flags in M^" ® M^. 

The space 3^ is an 0{p, (7)-homogcncous space. A minimal parabolic subgroup 
is the stabilizer of a point in 5". Let us give more explicit description of the 
minimal parabolic subgroup. 

For this let us consider the basis . . . , Vp, Wi, . . . , Wq-p, v'l, . . . ,v'pm MP(BR'' 
defined by 

1 1 

= -^{e, + eg+j); Vj = -^(e^ - eg+j); Wk = Sp+k (1.23) 

Then 

J(efe,efc) = l; J{fkjk) = '^ 

and the scalar products of all other pairs of basic vectors are zero. 

Denote by Lfe the subspace in W © generated by the basic vectors 
vi,. . . ,Vk- We denote by P C 0{p, q) the stabilizer in 0{p, q) of the isotropic 
flag 

L: LiC---CLp (1.24) 

The subgroup P is a minimal parabolic subgroup in 0{p, q) and 

3^^0{p,q)/P 

Elements of the parabolic subgroup P in the basis (1.23) have the form 

A * * \ 

C * (1.25) 
A*-^J 

where A is an upper triangular matrix and C € 0{q — p). 

Elements of the subgroup A (see Subsection 1.15) in new basis are diagonal 
matrices with eigenvalues 

f>*i p*p 1 1 f>-*i p-tp 

C- , . . . , C- , J-5 . . . , J-5 C- 5 . . . , O 

Remark. Let us change the order of basic elements (1.23) to vi,...,Vp 
,wi, . . . , Wq^p, v'p,. . . ,v'i. Then elements of the parabolic subgroup P will be 
upper triangular matrices. 

1.17. Spherical representations. Denote by Gr^ the space of all k- 
dimensional isotropic subspaces in M^'SM^. Consider the tautological embedding 
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of the flag space 5" to the product of the Grassmannians x^^-^Gr^ ( to each point 
V : Vi C ■ ■ ■ C Vp we assign the point (Vi, . . . , Vp) E Xfe=iGrfe). 

Consider the natural action of 0{p,q) on Gik- For g E G we denote by 
jk{g, V) the Jacobian of the transformation g at the point V G Gr^. By J{g, V) 
we denote the Jacobian of the transformation g on the flag space J'. 

Fix Si, . . . , Sp £ C. Assume sq — Sp+i — 0. We define the representation tt^ 
of the group 0{p, q) in the space of functions on 3^ by the formula 

M9)f iVi, ...^Vp)^ figVi, . . . , gVp)Jig, Vf/^ f{ Mg, T4)(«.--2.,+s,+,)/2 

Remark. The representation tt^ is a Harish- Chandra module^ and hence 
a topology in the space of functions on 3^ is not essential. For instance, we can 
consider the space L^{3^), the space of smooth functions C°°(3^), the space of 
distributions T>{3'), the space of hyperfunctions etc. 

Remark. Consider the (5-function (5^ supported at the point iL e 3^ (see 
(1.24)). It is easy to observe that the function is an eigenfunction of P and 
for a matrix g E P given by (1.25) we have 

p 

^s{9)SU^) = exp{^t,(s, - (q+p)/2+j)}6^{V) (1.26) 

where e*^ are the absolute values of the eigenvalues of the matrix A (see (1.25)). 
We want to define a canonical irreducible subquotient tTs in tTs- 
Remark. For generic s S C" the representation tTs is irreducible and hence 

Consider the function /o(V) = 1 on the space it is the unique iC- invariant 
function on J' (since 9^ is if -homogeneous) . Denote by S the cyclic span of /q. 
Denote by R the sum of all proper 0(p, (7)-submodules in S. 

Lemma 1.11. S. 

Proof. Indeed there is the unique K-fixed vector in S and this vector is 
cyclic. Hence, it can't be element of a proper submodule. Hence, a proper 
submodule in S hasn't K-tixed vector. Hence, R also hasn't K-tixed vectors 
and hence fo ^ R- Kl 

We define the 0(p, (7)-module tTs by 

TT^ = S/R 

Theorem 1.12.^ The representations tt^ are precisely all spherical repre- 
sentations ofO{p,q). Moreover 

TTs ~ TTs' iff there exists 7 € Dp such that 7s — s' 

^^This means (for instance, see th^^t the spectrum of the maximal compact subgroup 

K in the space of functions on 3^ has finite multiplicities, 
^^for instance, see |17T, Theorem 4.4.3 
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Hence, we can consider our Plancherel measure Va as a measure on CP / Dp. 
It will he more convenient for us to consider the Plancherel measure as a Dp- 
invariant measure on C or any measure on on CP whose Dp-average is Va- 

1.18. Unitary spherical representations. Lemma 1.13. Assume the 
representation tTs be unitary. Then for any j 

Resj=0 or lmsj=0 (1.27) 

Proof. The representation dual to tTs is 7r_s. The complex conjugate 
representation to tTs is tt^. If tTs is unitary, then the dual representation is 
equivalent to the complex conjugate representation. Hence, — s = 7s for some 
1 & Dp. Kl 

If si,...,Sp are pure imaginary, then the representation tt^ is unitary in 
These representations are called representations of the principal nonde- 
generate series. 

For some other values of s representations tt^. also are unitary, but scalar 
product in these cases is more complicated. 

Theorem 1.14. (see 4.8.1) Denote by p the vector 

{{q+p)/2-l,iq + p)/2 -2,...,{q- p)/2)) G W 

Denote by Q the convex polyhedron in with vertices jp where 7 £ Dp. Then 
for each unitary representation tt^ 

(Resi,...,Resp) e g (1.28) 

Moreover, spherical function of a spherical representation tt^ is bounded if and 
only if condition (1.28) holds. 

Our next purpose is to give the integral formula for the spherical functions 
in an explicit form. For this we must give another realization of G/K . 

1.19. Matrix wedges. First, consider the casep = q. Consider the matrix 
ball Bq.q(R). Consider the Cayley transform 

Cay : z ^—^ (1.29) 

1 + z 

Then the map Cay transfers the matrix ball Bg,g(M) to the wedge Wq consisting 
of matrices R satisfying the condition|^ 

i? + i?* > 

(where the notation Q > means that a matrix Q is positive definite). It is 
convenient to wright R in the form 

R = T + S where T = T* > 0: S = 



matrix satisfying the condition i? + i?' > is called dissipatv 
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The group 0{q, q) acts on Bg^q{R) and hence it acts on Wg,g. For description 
of the last action we consider the basis (1.23) in ® M'. In our case p = q, 
and hence the basic elements Wj are lacking. Hence, 0{q, q) becomes the group 

of real (9 + 9) x (g + g')-matrices having the form = ( ^ ^ j and satisfying 



the condition 

U b\ /O 1\ /a bV /O 1 



The group 0{q,q) acts on Wg by fractional linear transformations 
R ^ R^a] ■- (a + Rc)-^{b + Rd) 

In this model, the parabolic subgroup P C 0{q,q) becomes the group of real 
matrices having the form 

Q (jt-i^ ' where d is upper triangular (1.30) 

Hence, the parabolic subgroup acts on Wq by afBne transformations 

R^ a~'^Ra* + a~^b 

Wc emphasis that a^^b is a skew-symmetric matrix. 

We can easily wright the eigenfunctions of the group P on the wedge W, 

(Ji) = n det[T]5.-^^+«^-«^+i)/2 where 9^ = ■ ■ ■ = 6^.^ = I, 6^ = 

(1.31) 

Here the symbol [T]j denotes the left upper j x j block of the matrix T. 

Consider g & P given by formula (1.30). Let e*% . . . , e*" be the absolute 
values of the diagonal elements of the block a. Then 

q 

^s,,...,sM'^) = exp{J2isj - q + j)tj}^s,,...,s,{R) (1.32) 
Remark. Compare (1.32) and (1.26). 

1.20. Sections of wedges. Consider arbitrary group 0{p,q). Let us 

represent a point z G Bp ,j(R) as block p x {p + {q — p))-matrix z = {zi 22). 
Consider block {p+ {q — p)) x {p+ {q — p)) matrix 





Zl Z2 
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Thus, we realized the matrix ball Bp,g(M) as a submanifold of Bg_g(M). The 
image SWp,g of Bp,q(R) under the Cayley transform (1.29) is the set of {p+{q — 
p)) X (p + (g — p))-matrices R <E Wg having block structure 



R 



1 
Q H 



(1.33) 



The condition R + R* > for matrix (1.33) is equivalent to the condition 



(1.34) 



(spaces SWp^g are real sections of so-called Siegel domains of the second type, 
see [0) 

We will Wright matrices R G SWp.g in the form 



R 



(1 

\2L M + N 



The condition (1.34) can be represented in the form 



M ~ LL* >0 or 



1 L' 
L M 



> 



(1.35) 



The eigenf unctions of the parabolic subgroup P C 0{p, q) in this model are 
given by the formula 



*..,...,.p(^)-nd^t 





"^11 






= 1, 


dp = 



W det[M - LL 



j 



(1.36) 



The Berezin kernel (see Subsection 1.10) in the models W^, SWp^g is 
given by the formula 



La{Rl,R2) — 



det(i?i +i?|) 



Ldet(l + i?i)det(l + i?2)J 
This gives the following expression for the function 'Ba{R) in our coordinates 



det2" 



1 L 
L M 



det{l + M + Ny 
The 0(p, g)-invariant measure on SWp.g is 

rt\ -(P + 9)/2 



(1.37) 



det 



1 L 
L M 



dL dM dN 
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where dL, dM, dN are Lebesguc measures on the spaces of matrices. 

1.21. Canonical embedding of the spherical G-module tTs to the 
space C°°{G / K). First, we define the canonical intertwining operator Jg from 
TTs to C°°{G/K). This operator is uniquely defined by the following property 

Js : 5c ^ *s 

where P-eigenf unctions 5c,-, were defined in Subsections 1.17, 1.19-1.20. By 
the intertwining property we obtain 

Js5gc{R) = -^s{R^'^) 

and this defines the operator Js on all J-functions. Then we extend Jg by 
linearity and continuity to the whole space of distributions on J. 

Lemma 1.15. The operator Jg induces an embedding of the suhquotient tTs 
to C°°{G/K). 

Le us denote by dk the Haar measure on = 0(p) x 0(g). We assume that 

the measure of the whole group is 1. 

Proof. Let R, S be the same as in Subsection 1.17. Let Q C C°°{G/K) 
be a G-invariant closed subspace. Then for any function f G R, its average 

f^{R) = [ fi^dk 
JkeK 

is contained in Q. Hence, Q contains a if-invariant function. 

By this reason, Jg maps the submodule i? to (since R hasn't if- invariants). 
Assume that Jg is zero on S. Then Jg is an operator from tTs/S to C°°{G/ K). 
But the module t^s/S hasn't JT-invariant vectors. Hence, Jg is identical zero 
and this contradicts to its definition. lEl 

Obviously the if-fixed function /o = 1 on can be represented in the form 

/o(V) = / <5x:(kV)dk 

J^^eK 

Hence, its image under Jg is the JsT-average of '^g. This gives the integral formula 
for spherical function given in the next Subsection. 

1.22. Integral formula for spherical functions. Spherical functions are 
if -averages of P-eigenfunctions onG/K 

^g,,...,g,{R)= [ (iiM)d7r(k) (1.38) 

^keO(p)xO(g) 

Lemma 1.16. 

|*si,...,sp(t)| <^Resi,...,Resp(i) (1-39) 
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Proof is obvious. Kl 
1.23. Spherical transform. Let f{z) be a X-invariant function on G/K. 
Then the spherical transform of / is defined by the formula 



f{s) = / <i>.{z)f{z)dX{z) (1.40) 
Jg/k 

where A is the G- invariant measure on G/K. 

li f G n L^{G/K), then the Gindikin-Karpelevich inversion formula (see 
[0, 0, 0, §) is valid 

f{z)^C- I J{s)^s{z)d\{s)ds (1.41) 

where C is a known constant (see [0, formula (4.6.40)) and 5R(s) is the Gindikin- 
Karpelevich density. For G = 0{ja, q) it is given by the formula 

. ^ r((,-p)/2|,.)r((, p)/2-,.) ^ 

r(sfe)r(-sfe) 

-pr r(l(l + SI + Sk))T{\{l + .Si - Sfc))r(l(l - .s; + gfc))r(l(l - - Sfc)) 

liip r(i(.i + ..))r(i(,si - .fe))r(i(-,si + ..))r(i(-,si - .,)) 

(1.43) 

Remark. This expression is an elementary function. For instance, using 
the complement formula for F, we reduce factor (1.43) to the form 

n {si - sj)ta.mi{sk + si)ta.mi{sk- si) (1.44) 
If q — p is even, then (1.42) equals to 

p (g-p)/2-l 

n{ n (1-45) 

k=l T=0 

If {q — p) is odd, then (1.42) equals to 

P (?-p-3)/2 

[]{sfctan^Sfc W ((t + 1/2)2 -s2)| (^ ^g^ 

k=l T=0 

For pure imaginary s we can replace (1.42)-(1.43) by 

^| F((g-p)/2 + afc) ^ | F(l(l + ,s; + .Sfc))r(l(l + - .s^)) 

r(^..) ^Ji^J F(i(.i + ..))r(i(.i-.,)) 
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since Sk is imaginary. Nevertheless long expression (1.42)-(1.43) is more conve- 
nient for our calculations. 

1.24. Another formula for spherical transform. By integral formula 
for spherical functions (1.38) we can wright spherical transform (1.40) in the 
following form 

fis) = [ f{z)-^s{z)dX{z) (1.47) 
Jg/k 

1.25. Further structure of the paper. We want to obtain an expansion 
of the function "Baiz) in spherical functions. If jB^ e fl L'^{Bp^q{R)) (or 

a > p + q — 1), then it is sufficient to evaluate the spherical transform of the 
function 'Ba{z), and the Gindikin-Karpelevich inversion formula gives required 
expansion. 

In Section 3 wc evaluate the spherical transform of 25q;(z) using formula 
(1.47). The final result is given in Theorem 2.1. Then in Section 4 we con- 
struct the analytic continuation of our formula to arbitrary a. As result, we 
obtain an expansion of 25a in spherical functions. In Section 5 we prove positive 
definiteness of these spherical functions. 

D. Deformation of on Riemannian compact symmetric 
space and kernel representations of 0{p + q). 

This subject is a supplement to the main topic of the paper. 

1.26. The symmetric spaces V{p + q)/\J{p) x \J{q) and 0{p + q)/0{p) x 
0{q). Consider the group \J(p+q) consisting of all complex block {p+q) x (p+q)- 

matrices ^ satisfying the condition 

fa b\ fl 0\ fa bV _ fl 0\ 
\c d) \Q ij \c d) l) 

Consider the subgroup 0{p + q) C U(p + q) consisting of real matrices. 

Consider the Grassmannians Grp,g(C) and Grp^g(R) consisting of p-dimensional 
subspaces in and M*'+' respectively. Obviously, these Grassmannians are 
symmetric spaces 

Grp,,(C)=U(p + 9)/U(p) xU(9) 
Grp,,(M)=0(p + g)/0(p)xO(g) 

Denote by Matp,g(C) (resp. Matp,g(lR)) the space of all p x gr-matrices over 
C (resp. over M). For any z G Matp,, we define its graph GRAPH^ c Grp_g. 
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Obviously the map z ^ GRAPHz is an embedding of Matp,g Grp,g and the 
image of the embedding is dense in Gip,,. 

In the coordinate z E Matp^g, the action of the group lJ{p + q) on Grassman- 
nian is given by the formula 

z ^ z^s^ = {a + zc)-\b + zd) (1.48) 

coinciding with formula (1.7). 

1.27. Representations T_„. Fix n = 0,1,2,.... Denote by (Pa{z) the 
polynomial on Matp.g given by the formula 

Va{z) = det(l + a*z)"'; where a € Matp,, 

Denote by -ff_„ the linear span of all polynomials (fa{z). Obviously the space 
H-n is finite dimensional (since degree of the polynomial <fa{z) is pn). 

Consider the action of the group \J{p,q) in the space -ff_„ given by the 
formula 

^-"(c 2) =/((a + 2c)-i(6 + z(i))det(a + zc)" (1.49) 

coinciding with formula (1.10). It is easy to check, that the transformations 
T-n{g) preserve the space -ff_„. 

Consider the scalar product in given by 

< fi{z), f2{z) >_„= Cn I /i(0)Mi) det(l + z*z)-''-P-Uz 

where the normalization constant C„ is defined by the condition < 1,1 >_«= 1. 

It is easy to check that the operators T-n{g) are unitary with respect to this 
scalar product and 

<ipa,(Pb>-n^dct{l + a*b)'^ (1.50) 

Hence, the (finite dimensional) hilbert space is the hilbert space H° defined 
by the positive definite kernel 

L_„(a,6) =det(l + a*6)"; a,6GMatp,, (1.51) 

1.28. Kernel representations of 0(p + q). The kernel representation 
T-n of the group 0{p + q) is the restriction of the representation T_„ to the 
subgroup 0{p + q). 

The also define the marked vector S 

S : f{z) = 1 
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1.29. Limit as n ^ oo. Let us consider the kernel 

det(l + 

'^"^^'"^ " det(l + z*z)"/2det(l + ii*u)"/2 

on Matp,5(M). By Lemma 1.5.e) the kernel M_„ is positive definite. A simple 
calculation shows, that the kernel is 0{p + g)-invariant. Consider the hilbert 
space H°[M^n]- The operator 

A/(z) = det(l + z*z)"/VW 

defines the canonical unitary 0(p + g)-intertwining operator H°[La] H°[Ma]. 

The arguments given in Subsection L13 show that a natural limit of the 
spaces H-n as n ^ oo is 

L'{0{p + q)/0{p) X 0(g)) 

1.30. Preliminary remarks on the Plancherel formula. By Subsection 
1.7 the matrix element 

S_„(.g) =<T_„(.g)S,S>ff_„ 

is a function on 0{p + q)/0(p) x 0{q) ~ Grp ,j(M). In the coordinate z €E 
Matp^g(M) it is given by 

•B_„(z) =det(l + zz*)-"/2 

Denote by 0{p + q)gpf^ the set of all irreducible representations of 0(p + q) 
having an 0{p) x 0((7)-invariant vector {spherical vector); description of this 
(countable) set is given by Helgason theorem, Theorem 5.4.1. By Hp we 

denote the space of a spherical representation p £ 0(p + 9)sp/i- Denote by 
the spherical vector in Hp having unit length. 

Arguments given in Subsections 1.13-1.14 show, that the decomposition of 
T_„ in irreducible representations has the form 

r_„(5) = Pig) 

p6A„ 

where A„ is a finite subset in 0{p + q)spi^- 

The scalar product in 0pg^^ Hp has the form 

peA„ peA„ peA„ 

where Vp,Wp € Hp and z/^ are positive constants. The formula (1.52) is called 
the Plancherel formula. 
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We normalize the constants i^" by the assumption 



the image of S in 



is 



peA. 



pSA 



The constants i'" are evaluated in Section 2 as a corollary of the Plancherel 
formula for kernel-representations of 0(p, q). 



E. An interpolation between 



L^(0(p,q))/0(p) X 0(q)) and L^(0(p + q))/0(p) x 0(q)) ? 



The purpose of the Section is a formulation of a strange problem. 

1.31. General representations T^. Denote by Hol(Bpq) the space of 
holomorphic functions in Bp^(C) equipped with topology of the uniform con- 
vergence on compacts. 

Consider arbitrary a G C and consider the action of \J{p,q) in Hol{Bp,q) 
given by the formula 



Denote by J{° the cyclic span of the function f{z) = 1. Denote by Ta the 
restriction of Ta to . It is easy to observe that Ta is an irreducible Harish- 
Chandra module. 

Let us denote by H^"^ the space of polynomials contained in J{° . Consider 
the action of the Lie algebra u{p, q) in Hl"\ The space i/^™ is an irreducible 
u{p, (7)-module with a highest weight. For a g R there exists the unique u(p, q)- 
invariant hermitian form in i/^™ (it is called Shapovalov forrif^. In general 
this form is indefinite. 

If a S M satisfies Berczin conditions (1.9), then the Shapovalov form is pos- 
itive definite. It coincides with the Berezin scalar product, and representations 
Ta coincides with representations Ta constructed in Subsection 1.10. If a is a 
negative integer, then Ta is finite dimensional. By the unitary Weyl trick, there 
is no difference between finite dimensional representations of U(p, g), holomor- 
phic finite dimensional representations of GL(p + q,C) and finite dimensional 
representations of\J{p + q). The representations Ta for negative integer a differs 
from the representations r_„ from Subsection 1.27 by a nonessential change of 
notations. 

^^Its definition for highest weight modules of various group is uniform, see for instance |32l. 




(1.53) 



27 



1.32. Nonunitary kernel representations of 0{p,q)? Consider tiie 
restriction Ta of Ta to the subgroup 0(p, q). It is a well-defined representation 
of the group 0{p, q) in the space ?{° . 

We have seen that 

lim T^^L^Oip,q))/Oip)xO{q)) 
lim T„ ~ L^{0ip + q))/0ip) X 0(g)) 

n — > — QO 

It seems that the Plancherel formula (2.5)-(2.15) gives the decomposition of 
the kernel representation Ta for any complex a. Unfortunately it is a result of 
'mathematical physics level' . This is the solution of a problem which hasn't a 
satisfactory formulation (since the definition of the abstract Plancherel formula 
doesn't exist for nonunitary representations). 

Remark. For the case p ~ 1 the space ?{° equipped with the Shapovalov 
form is a Pontryagin space|^ and in this case our Plancherel formula is really 
the Plancherel formula for arbitrary real a. 

The questions of this type are discussed for a long time and they arise to deep 
Molchanov work |27j(1980) containing the Plancherel decomposition of tensor 
products of unitary representations of SL(2,M) (see also [|j and references in 
this paper). It was clear, that Molchanov formula give formal interpolation 
between tensor products of unitary representations of SL(2, R) and tensor prod- 
ucts of finite dimensional representations. Unfortunately before our time a quite 
satisfactory group-theoretical interpretation of this interpolation doesn't exist. 

2. Formulation of results. 

2.1. Large a. Theorem 2.1. Let a > {p + q)/2 — 1. Then the spectrum of 
the kernel-representation Ta is supported by nondegenerate principal series and 

■^"^This means that negative inertia index of the Shapovalov form is finite. 
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the Plancherel decomposition is given by the formula 
p 



ncosh-«t,=C.2«^^^— (2.1) 



k=l 



[ n(rd(a-(p + «)/2 + l + Sfc))r(i(a-(p + g)/2 + l-Sfe))| x 

(2.2) 

rr r((g-p)/2 + sfc)r((g-p)/2-sfe) 

M r(..)r(-s.) ^'-'^ 

j-j r(i(i + s; + sfc))r(i(i + Si - sfe))r(i(i - + sfe))r(i(i - - s^)) 



X 



(2.4) 

X ^si,...,sp{ti, ■ ■ ■ ,tp) dsids2 . . . dsp (2.5) 

where C is a constant. 

Remark. Factor (2.3)-(2.4) is the Gindikin-Karpelevich density, it is an 
elementary function, see (1.44)-(1.46). 

2.2. Analytic formula for arbitrary a. Fix m = 0,1,..., p. Consider 
nonnegative integers 

Ml ^ M2 ^ • • • ^ Mm 



satisfying the condition 



a + 2um + m< ^{p + q) 



(if m = 0, then a collection {u} is empty). 

Theorem 2.2. Let p ^ q and a he arbitrary, or p = q and a G R\ 
{l,2,...,p- 1}. Then 



cosh °'tj=C- ^ Em{a,u) X 

m; 

ui ^ . . . < j{p + q) - ^ - 

{a,u;s)^Krn{s) ^Q — (p+q)/2+l+2ui,...,a — (p+q)/2+m+2tt^,s^^i,...,s^ (^Ij ■ ■ ■ j ^f)) X 

X dSm+1 ...dsp (2.6) 
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where C is the same as above 

p 



E^ia,u) = i2.r^2-Jl^,^-^^. (2.7) 
-pi- (-a + i(p + g) - 2ur - r)r(a - p + r + 2u^)r(-a + g - r - 2ur) 

j^Jl {Ur - Ur-l)lT{-a + ^{p + q) - T + 1 - Ur - Ur-l) 

(2.8) 

X Yl_ ^{-a + ^{p + q) - ^{t + a) - U„ - Ur)i^{T - a) + Ur - Ua) X 

(2.9) 

r(i(T — a + 1) + Ur — Ua)T{~a + i(p + q) — ^(t + cj) - u,- — Uo- + 5) ' 



X 



Ur) J 



r(5(''' - cr) + - u^_i)r(-a + ^{p + q) - ^(cr + r) - - Ut) 

(2.10) 



Ym{a,u;s) = 

p 

= n {r(^(a-i(p + g)+m + l + Sfc))r(i(a-i(p + g) + m+l-Sfe))} X 

fc=m+l 

(2.11) 

X n I (|(-a+5(P + a)-r-2w^ + Sfe))(i(-a+i(p + g)-T-2«;^-Sfc)) X 

r^m; k>m \ 

(2.12) 

^ r(|(-a + ^(p + g) - (t - 1) - 2m^ + 5fc)r(l(-a + ^{p + g) - (r - 1) - 2u^ - s^) ' 
^ r(i(-a + i(p + g) - r + 2 + 2«;^_i + Sfe)r(i(-a + i(p + g)-r + 2 + 2«;^_i - Sk) 

(2.13) 



T-r r(i(l +SI + Sfc))r(l(l + - Sfc))r(l(l -Si+ 5fc))r(i(l - S; - Sfe)) 

(2.15) 

Remarks, a) The factor 9lm(s) is an elementary function. 

b) More convenient notations are used in Section 4 (sec 4.13). 

c) The formula, which is not so explicit, but short is given in Section 6. 
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Remark. The summand corresponding m = coincides with integral (2.1)- 
(2.5). For summands corresponding m = p, the integration is given by one point 
set and hence these summands are spherical functions with some coefficients. 

2.3. The case a = p— l,p— 2, . . . , 1. In this case some summands disappear. 

Proposition 2.3. Let a = p — h where /i < p. Then the factor Em{a, u) is 
nonzero if and only if 

m ^ h; Ui = U2 = ■ ■ ■ = Uh = 



Proof. Vanishing of Em{ct,u) is completely defined by a behavior of the 
factor 

The denominator has a pole of order h a,t a = p — h. If the fraction is non- 
vanishing, then the numerator has a pole of order h at the same point, boxtimes 

2.4. The case a = —1, —2, —3, Assume Em{a, u) ^ 0. The denomina- 
tor of (2.16) has a pole of order p in a. Hence, the numerator also has a pole of 
order p. Hence, 

m = p 

This means that all integrals in Plancherel formula (2.6) vanish and we have 
only finite sum of spherical functions with some coeSicients. The coefficient 
Ejn{a,u) is nonzero iff 

m + 2?i„, ^ — a 

2.5. The Plancherel formula for the kernel representations of 

0{p,q). 

Theorem 2.4. Let a satisfies Berezin conditions (1.9). Then 
a)if Em{oi,u) 7^ (see Subsection 2.3), then all spherical functions 

^a — (p+q)/2+l+2ui ,...,a— (p+g)/2+m+ 

are positive definite. 

h)formula (2.6)-(2.15) is really the Plancherel formula 

2.6. The Plancherel formula for kernel-representations of 0{p + q). 

For a negative integer a = —n (see Subsection 2.4 above) formula (2.6) gives the 
expansion of det(l — z*z)" in 0{p) x 0((3')-spherical functiouK of 0{p, q) and this 
is equivalent to the Plancherel formula for the kernel representations oi 0{p + q). 

2.7. The case of indefinite Shapovalov form. For noninteger a < p—1 
we obtain the problem discussed in Section I.E. 
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3. B-function of the space OijPil) l^ip) ^ 0(0') 



In this section we construct a matrix imitation of the B-integral 

B(x,y) = / — r — —dt 

for the symmetric spaces 0{p, q)/0{p)xO{q). For symmetric spaces GL(n, K) /U(n, K) 
the B-integrals were defined by Gindikin [0 (see also exposition in [||), for other 
symmetric spaces B-integrals were derived in [^5| . 
3.1. B-integral. Let 

Ai, . . . , Xp, ai, . . . ,(jp e C 

We also assume 

Ap+i ~ o-p+i = 

Let SWp^5 be the section of wedge defined in Subsection 1.20. 
Theorem 3.1. Let Xk, Uk satisfy the inequalities 

i(g + fc)/2 + l<Afc<c7fc-i(p-/c) (3.1) 

Then 

f det[M-LL*]^^"^^+^ , ^ , 

/ n dct(Af - LL*)-^P+''^/^dMdNdL = 



(3.2) 



det 



n 



1 L* 
L M 



M - LL* >0 
N = -iV* 



/^^idet[l + M + iV]J^' "^+^ 



"^'^^^ det i ^ 



t \ -(P+9)/2 



dLdMdN 



n 

fe=i 



.k-(,-,)/2-i r(Afc - (g + fc)/2 + l)r(afc -Xk-jp- k)/2) 

r(crfc -p + k) 



(3.3) 
(3.4) 



The proof of the Theorem is given in Subsections 3.2-3.6. 

Remark. For p ^ q we have L ~ and integral (3.2)-(3.3) has more simple 
form, see (0.5). In this case the calculation given below also is simpler. The 
main simplification is the expression for matrix (3.13): the first block row and 
the first block column are lacked. 
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Remark. We have M = Af* > 0, iV = -A^*. Hence, 

det(l + M + 7V) > 

Indeed, for any v £ we have Kev{M + N)v* = vMv* > 0. Hence, the 
eigenvalues Xj oi M + N satisfy the condition Re Aj > 0. Hence, the eigenvalues 
of 1 + Af + arc nonzero. 

Remark. Hua Loo Keng in |2l| evaluated the integralsH 



det(l - zz*ydz 



(3.5) 



Cayley transform reduces the Hua integral to the following partial case of our 
integral 

'1 L*^ ' 
.L M , 



const 



det 



det(l + M + Af)2^ 



dL dM dN 



Our calculation in this case is not homotopic to Hua calculations. 

3.2. Replacement of notations. Firstly, we call to mind the standard 
formula (sec [|l^) for determinant of block (to + n) x (m + n)-matrix 



det 



A B 
C D 



= detA-Aei{D-BA-^C) 



(3.6) 



Let us represent M, N as block {{p — 1) + 1) x {{p — 1) + 1) matrices, and L 
as a block {{p — 1) + 1) x (g — p) matrix: 



M 



P 9* 
q r 

Then for j p — 1 

1 

L M 



N : 



A -6* 
b 



L 



H 
I 



coincides with 



g-p+j 

[1 + M + N]j coincides with 



1 H* 
H P 



[1 + P + A]j 



and by (3.6) 

dct 



1 

L M 



det 





( 1 




= dct 


H 


P 






q 










P 


)■ 



1 H* 
H P 



I* 



det(l + M + N)= det(l + F+A) ■{l + r-{q + h){l + P+A)-^{q^ - 6*)) 

^^Hua integrals also can be reduced to the Selberg B-integrals by integration over K\G/ K . 
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(1 L* \ 

By the Sylvester criterion the condition ( ^ Ml ^ ^ ^^^^ i^-^^)) 
notations has the form 

H P )>''^ •■-(' P )"('.) >° P-') 

By the remark given in Subsection 3.1 

det(l + P + A)>0 
3.3. Substitution. Let us replace the variable r to the variable 



u = r ■ 



1 H' y Y I' 

HP) \ 



(all other variables are the same). By (3.7) we have u > 0. The Jacobian of the 
substitution is 1. Our integral converts to the form 

j dP dA dH ^{A, P, H) x (3.8) 



X / uV-(P+9)/2 



i+u+{i o( ) 7 )+ (3-9) 



1 




H 


P 


i 






6* i 



(3.10) 

where 



p-2 det 



1 H* 
H P 



det ( ^ p 



p m i-p+j \ / 

" 7=1 det[l + P + A]J^-''^+^ det(l + P + A)<^n-i 

(3.11) 

is an expression independent on u, b, I, q. 

Firstly, we want to evaluate interior integral (3.9)-(3.10) 

3.4. Transformation of the integrand. Denote by S the expression 

S=l+P+A 

Let us represent the expression in the curly brackets in (3.10) in the form 

I' 



|l + «+( I q 6 )X f jl 



(3.12) 
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where 



X 



1 iJ* 
H P 









-5-1 



V 



( 5-1 



(3.13) 



(we wright a block matrix whose elements are block matrices itself). The last 
summand in the curly brackets is a quadratic form in the variables 6, q, I. But 
the matrix X is not symmetric and it is more natural to re-wright expression 
(3.12) in the form 



|l-H«+( / q b)\{X + X') 




(3.14) 



3.5. Separation of variables. 
Lemma 3.2. 



det( |(X + X*) 



Proof, det (i(X -h X*)) 



= det 









{I 







det 



1 ff* 
H P 



det(l+P + ^)" 



1 Qt-l 1 Q-1 

2'^ ~ 2*^ 



V 



( i5-i - i5*-i ) 



45-1 + 15*-! 
15-1 + i5*-i 



/ 



Adding the third row to the second row and the third column to the second 
column, we obtain 



det 



1 

H P 







(l + P-A)-i 

V(o (i+p+^)-i ) i(i + p+A)-i + i(i + p-A)-i y 



Formula (3.6) reduces the determinant to the form 

1 iJ* ^ 
H P 

I H* \ ( 



det 



• det 



1 



^)-i + i(l + P-A)-i 



-( (l + P + ^)-i ) 
= det 



H P 



1 iJ* 
H P 



(l + P-A)-i 

-1 

det(l + P + A)-i det(l + P - A)'^ x 
X det(i(l + P-A) + \{1 + P + A)-P) 
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The last factor is 1. We also observe 

{1 + P + AY = 1+P-A 

and hence their determinants coincides. Kl 
Lemma 3.3. X + X'^ > 0. 
Proof. In the identity 

det(l + M + N) = det(l+P+A) ■[l + r-{q + b){l+P+A)-\q* - 6*)] 

we have det(l + M + TV) > 0, det(l + P + A) > 0. Hence, the factor in the 
square brackets is positive. Hence, expression (3.12) is positive for all u > 0, 
and all g, b, I. Quantity (3.12) coincides with quantity (3.14). Hence, the matrix 
X + X'^ is nonnegative defined. By Lemma 3.2 its determinant is nonzero and 

we obtain the required statement. K 

Consider the linear substitution 

{ I q b) ^\{x + x*) = he w-p e M.p-'^ ® Rp-i 

to interior integral (3.9)-(3.10). Its Jacobian is 



detf^ pj •det(l + P + A) 



and hence the interior integral coverts to the form 



det ( ^ p ) • det(l + P + A) x (3.15) 





y/2 










p-2 





X J M^^-^f+«J/^{l + M+|/i|^} ^dudh (3.16) 

The first factor (3.15) adds to the product S(A, P, H) (see (3.11)) and we reduce 
our B-integral (3.3) to the product of the integrals 



P-HtH>0,A=-A- 



1 H^^'" 



E{A,P,H)det( ^ p] ■ det{l + P + A)dAdPdH X 



X J W^''-(f+9)/2{l + y+|/j|2} ''"dudh 

u>o, /ieK«+p-2 
Let us denote B-integral (3.3) by 

Ip,g{oii,---,ap;ai,...,ap) (3.17) 
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and let us denote factor (3.16) by Jp,q{c(p', Op). We obtain the following recurrent 
identity 

Ip,q{ai, . . . ,ap\(Ji, . . . ,ap) = 

= Ip-i,q-\{oii, . . . , ap-2, ap-1 — 5; (Ti, . . . , iJp_2, CTp-i — l)Jp,g(ap; CTj,) 

3.6. Evaluation of Jp^q{ap; ap). This problem is trivial. Firstly, we con- 
sider spherical coordinates in W~^'^~'^ in the variable h. Then Jp^g{ap;ap) con- 
verts to the form 

2^(p+«)/2- 



r((p + g)/2-i) 



1) Ju>0 Jr>0 

The substitution v = r'^ reduces our integral to a special case of the Dirichlet 
B-integral 

u^-^v^-^ , , T(a)T(b)T(c) 

— , , . , du dv = — ; r 

u>o,v>o (1 + u + v)^+'>+^ r{a + b + c) 

This completes the proof of Theorem 3.1. 

3.7. Spherical transform of "Ba- 

Corollary 3.4. Let a > p + q— 1. Then spherical transform of "Ba is 
ff 41 ^ n - 5(p + 9) + 1 + sfc))r(M" - ^(P + ?) + 1 - sk)) 

(3.18) 

Proof. The function "Ba is given by the formula (1.37). By Subsection 1.24 
we must evaluate the integral 

/ 'Ba{z)'^s{z)dX{z) 
Jg/k 

But the integral is a special case of our B-integral. 

3.8. Proof of Theorem 2.1. By the Gindikin-Karpelevich inversion for- 
mula and Corollary 3.4 we obtain the statement of the theorem for a > {p+q) — l. 

For a > {p + q)/2 — 1 the statement of the Theorem follows from trivial 
Lemma 4.1 proved below. 



4. Formal analytic continuation. 



We proved the Plancherel formula (2.1)-(2.5) for large values of the parame- 
ter a. Its left part ncosh~"(tj) depends analj^tically on a € C. The integrand 
in the right part has singularities on the lines 

Rea = ^{p + q) -l-2x; where x= 0,1,2,... (4.1) 
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Thus, the right part of formula (2.1)-(2.5) may be nonalytic for these values of 
a. 

Our next purpose is to construct the analytic continuation of the right part 
to arbitrary complex a. 

4.1. Analyticity. Let us denote the right part of the formula (2.1)-(2.5) 

by 

5(a) := J(a; t) = E{a) [ r(a; s)fR(s)$,(t) ds (4.2) 

where the meromorphic factor E{a) is given by formula (2.1), the factor Y{a; s) 
is defined by (2.2) and lH(s) is Gindikin-Karpelevich density (2.3)-(2.4). In 
this Section we fix the variable t and we omit the argument t from the notation 
dia;t). 

Consider domains XIq, IIi, . . . in C defined by 

Ho : Re a > i (p + g) - 1 

Hfe : i(p + g) - 1 - 2fc < Rea < i(p + q) - 1 - 2(fc- 1) where fc > 

Lemma 4.1. The function diet) is an analytical function on for all 
= 0,1,2.... 

Proof, a) Convergence of integral (4.2). First, the Gindikin-Karpelevich 
factor D\{s) has a polynomial growth in s, see formulas (1.44)-(1.46). 
By the formula (see ]l9|,1.18.6) 

|r(a + zy)| = (2^)i/2|y|-i/2exp{i^|y|}(l + o(l)); |y| ^ oo (4.3) 

the factor Y{a; s) exponentially decreases. 

A spherical function $s (t) is a spherical function of an unitary representation 
and hence we have |4's(i)| ^ 

Hence, the integrand exponentially decreases and the integral absolutely 
converges. 

h)Existence of -^^ia). It is sufficient to prove uniform convergence of the 
integral 



/ ■^Y{a;s)m{s)<i>,{t)ds 



(4.4) 



in small neighborhood of a fixed point a. For this we needs in uniformity by a 
of o(l) in (4.3). In fact, the asymptotics is really uniform but formally we have 
no possibility to refer to [l9[| . Formula (4.3) is derived from the Binet formula 
(see ra,(1.9.4)) 



1 1 r 

Inr(z) = (z )lnz - z + - ln(27r) + / 

2 2 7o 



1 1 1 



e* - 1 t 2 



t-^e-^^dt 



^^For following inductive steps this arguments must be replaced by inequality (1.39) 
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This formula easily implies an uniform estimate of the form 

|r(a + iy)\ < const • exp(— (^tt — £)|t/|); |o — a| < S 
The Cauchy integral for derivative 

-27rif{z) = / j'^^dz 

implies the same estimate for derivative of F-function. 

We observe that integrand (4.4) is dominated by some function having the 
form 

Pis)eM-b^\sj\) 

where P{s) is a polynomial and & > 0. 

Thus, the function 5^(a) has a derivative in the complex variable a and this 
complets the proof. Kl 

Lemma 4.2 Let p ^ q. Then the function ^{a) is continuous on the line 

Proof. Let h = i(p + g) — 1 — 2x be one of our singular points. The 
singularity of the integrand near this point has the form 

^""^t-fi fa % ,2 n (4-«f)^(i+o(i)) 
fe=i ^> i^k<Kp 

Remind that are pure imaginary. Hence, the integrand is bounded in a 
neighborhood of the point a = h, s = 0. As we have seen in the previous proof, 
the integrand has an integrable majorant in a domain |sfe| > A, \a — h\ < e 
{a E M). By Lebesgue theorem about dominant convergence, expression (4.2) 
is continuous at the point a = h. Kl 

Remark. The function ^{a) is continuous at the real points a= ^{p + q) — 
1 — 2k but it is not smooth at these points. 

We denote the restriction of the function ^{a) to the domain by 

4.2. Analytic continuation of i?>i(a) through a point of a line Rea = 

^{p + q) — 1 — 2k. The following lemma is main in this Section. Its proof is 
given in Subsections 4.2-4.5. 

Lemma 4.3. Let ao satisfies the condition 

Re ao = 5 (p + g) — 1 — 2x; Im ao 7^ 

Then 
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a) the function ^>c{a) admits the analytic continuation to some small neigh- 
borhood 

Os: \a-ao\<S where 5 < min{l/1000, | Ima|/1000} (4.5) 

of the point 

b) for any 

a e n n^+1 

we have 

WW-M«) = ^nf(d7TT)^ ^'-'^ 



{-a + + q) --l-x- \)Y\a - + 1 + 2^)r(-n + ry - 1 - 2>^) 

X (4.7) 



r(-a+i(p + g)-x) 
/ n r(i(a-i(p + 9) + 2 + 2x±Sfc)) X (4.8) 



1 2^fe^p,± 

(i(-a + i(p + g) - 1 ± ,sfc))r(i(-a + i(p + g) - 2x± Sfc)) 



n 



X ■ ■ ^2 



2^.^p,± r(i(-a+i(p + g) + l±s,)) 

(4.9) 

T-r r(i(5-p)±Sfc) 

TT r(i(i + afc±50)r(i(i-.fe±aO 

^Ji,. r(|(..±.or(i(-..±.0 ^'-''^ 

X $a-(p+g)/2+l+2^,S2,.-,Sp(*)'^«2 . • . rfSp (4.12) 

In the last formula we use the following notation 

]Jr(a±s) :=r(a + s)r(a-s) (4.13) 



4.3. Existence of the analytic continuations. Let us represent expres- 
sion (4.2) (or (2.1)-(2.5)) for ^^(a) in the form 

Eipc) J l[T{^{a-^{p + q) + l±Sk))-^R{s)^s{t)ds 

Let El > £2 ■■■>■■■> £p ^ he very small (for instance ei < 6/10 where 6 
was defined in (4.5)). Consider the function 
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OsQ 


ao 


Ti2 


Hi 


Ho 

(p + g)/2-l 













Picture 1. The complex plane a. The lines a = {p + q)/2 — 1 — 2>c and the 
domains 11^ 



d>c{a;e) = E{a) [ T\r{l{a + - i(p + g) + 1 ± Sk)) ■ 9^(s)$«(t)ds 

(4.14) 

in the domain 

ni : -2x< Re(a- ^(p + g) + l) < -2(x- 1) -£i 

Lemma 4.4. a) The function {a; e) admits the holomorphic continuation 
to the domain Os {see (4.5)). 

h)The functions \die{oi;£)\ in 0$ are bounded by a constant independent on 

e. 

Proof, a) The factor 9l(s) is holomorphic in the domain |Re(sj)| < 1/4 
and its poles are very far from the contour L which is described below. 
Consider 

aeOsnU% (4.15) 

Then integrand (4.14) has poles on hyperplanes 

Sk = ±{^{p + q) — I — a — 2u — Sk); where u = 0, 1, . . . 

lfu = K, then the poles are lying near points ± Imao) on Picture 2 the poles are 
marked as black circles. The arrows show the direction of their motion if Re a 
decreases. The white circles show rough position of the poles than a € OsnW^_^_i 
Consider the contour Lk on the complex plane Sfc G C given by Picture 2. 
Let £ C be the product of the contours Lk- Obviously, for a G Oa, we can 
replace the integration over iRP in the formula (4.14) by the integration over £j. 
But the integral 

/ = / ll^{l{c^ + Sk-hiP + l) + ^±^k))-'yi{s)^s{t)ds 
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Ri 



Si 



Q 



Ri 



Picture 2. The contour Lk = -Ri U 5*1 U Q U 5*2 U i?2 on the complex plane Sk- 
The centers of the semicircles 81,82 are ± Imao; the radius of the semicircles 
is 106. 
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obviously is holomorphic with respect to a in the domain (indeed, the surface 
a doesn't intersect the singularities and the integrand exponentially decreases 

as |s| 00). 

b) Consider the parameter 9^. := Imsj. on the contour L^.. 
Lemma 4.5. There exist constants A = A{t), N such that 

I n r(i (a + efe - i (p + 9) + 1 ± Sfc)) • (i) I < ^ n (1 + 1^,- 1 )^ exp(-7r|e,- 1 ) 

for all (.si. . . . , Sp) S £. 

Proof of Lemma 4.5. Let us estimate all factors in the left part of the 
inequality. 

a) The Gindikin-Karpelevich factor ^R{s). By the formulas (1.45)-(1.46) for 
any imaginary Sk we have 



r(i(g-p) + 5fe)r(i(q-p)-sfc) 



r(sfe)r(-sfe) 



< const • (1 + lOklf^i-P^ 



The same expression is bounded on the scmi-circIcs Si,S2- 
We also must estimate the factor (1.44). Firstly, 



n(4-sf)K const Ha + 



fe=l k 

Secondly, let us estimate the factors 

tan(7r(sfe ± S;)) 

of (1.44). If Sfe, ,s; arc imaginary, then | tan(7r(sj. ± s;))| < 1. If s^, s; € Si, S2, 
then this expression is bounded (since 5*1 , ^2 are compact sets) . Let Sk be 
imaginary and si G Si, S2- Then we obtain a value having the form | tan(x+iy)| 
where x,y gR, \x\ < lOwd. Then 

I tan.T + taniy , 

tan(x + iy) — —\ < tana;+taniw < tan a; +1 ^ tan(107ro) + l 

' 1 + tan X tan ly ' 

b) The T -factor Y{a; s). By formula (4.3) for imaginary Sk we have 
|r(i(a + £fe- i(p + g) + l + Sfe))r(i(a + £fe- i(p + g) + l-sfe)| < 

< const • (1 + |^^|)Hea+e.-i(p+,) + l exp(-7r|^fe|) 

For Sk € Si,S2 the same expression is bounded (but very large). 

c) Spherical functions ^s{t)- By estimation (1.39) we have 

\^s{t)\ < ^Res{t) < max$„,...,^^(i) 
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where maximum is given over all real vectors (n, . . . , r^) satisfying the condition 
\rj\ < 10^. Hence, for a fixed t the spherical function in integrand is dominated 

by a constant. 

This completes the proof of the Lemma 4.5. IE 
Now we can complete the proof of Lemma 4.4. b). By Lemma 4.5 we have 

\d^{a;e)\^A{t) [ TT [(1 + |^,|)^ exp(-7r|e,|)] TT x(^i)rf^i • • • rf^p 

there the function x(^) is given by the formula 



if 



ImofQ — 1^1 
Imao — 1^1 



^ 10(5 
^ 10(5 



d0 I (100(52- (Imao- 0)2)-i/2 if 

Hence 

\d.ia;e)\ ^ Aitf (/"(! + I^l)"^ exp(-7r|0|)x(e)d^ 

This completes the proof of uniform boundedness of the functions 'S>c{oi;s) for 
a fixed t (Lemma 4.4b). Kl 

Now we are ready to prove existence of the analytic continuation of the 
function 'S^- 

Proof of Lemma 4.3.b. Let us denote by e/n the vector {si/n, . . . , Sp/n). 
Consider the sequence of functions 

in the circle 0^. Since the functions gn{oi) arc uniformly bounded, by Montel 
theorem there exists a subsequence which is uniformly convergent on each 
smaller circle. Let g{a) be its limit. By Weierstrass theorem g{a) is holomorphic 
in Og. It remains to notice that 

lim ^^{a;e/n) = dx{a) 

n — ^oo 

for a e Hj^n O^. Hence, g{a) is the analytic continuation of 5^3<(q) to the circle 
O5. 

4.4. Forcing of poles. First, we want to obtain an explicit formula for the 

analytic continuation of ^^^{a; e) to the domain nj^_|_]^. 

Let the contours be the same as above. Let iRk be the imaginary axis 
on the complex plane Sk- Consider the surface 

Lk = iMi X • • • X iRk-i X Lfc X • • • X Lp c 
We have £1 = = iW. 
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Consider a e O5 n H^^+i. Then 
d^+i{a;e)=E{a) J [] r(i(a + Sfe - i(p + 9) + 1 ± s^)) • $n(s)$«(i)ds 



fe,± 



(4.16) 



5>.(a; s) = E{a) / [] r(i(a + Efc - i(p + g) + 1 ± su)) ■ ^{s)^s{t)ds (4.17) 
/ fe,± 



Hence, 



n 



Looking to Picture 2 we observe 

/ 

Sff+i € -t'CT+i) ■ ■ ■ jCTp € Lp 



Res 

s<,=a+£„ - i (p+g) + l+2>i 



Res 

T= — «-£<, + 1 (p+9)-l-23<- 



rfsi . . . dSa-idSa+1 ■ ■ ■ dsp 



(4.18) 

(4.19) 
(4.20) 

(4.21) 



The integrand in (4.16)-(4.17) is an even function in Scr and hence two residues 
in (4.20)-(4.21) differ only by sign. The order of poles 

s„ = ±{a + Sa-l{p + q) + l + 2x) 

of the integrand is 1 and hence the residues can be evaluated by a simple sub- 
stitution 

Res = H„{a, e, s) := 

- 1 (p+g)+l+23< 



E{a) 



r(l(a + £^-|(p + g) + l-s^)) 
s<,-a-£<, + |(p + g)-l-2x 



=a+£„-i(p+q) + l+2^ 

(4.22) 



In this way, we reduce sum (4.18) to 



p I- 



Hcr{a, e, s)dsi . . . ds^r-idscr+i ■ ■ ■ dSp 



L X---XiR<,_i Xio-i-i X---Xip 
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We obtain an expression for ^3t+i{a;s) — 5>i(Q:;e)- Unfortunately the do- 
mains of integration is yet complicated. By this reason we apply transformation 
(4.18) to each summand in the last expression. Wc obtain p{p— l)/2 additional 
summands which are integrals over (p — 2)-dimensional surfaces. Each integral 
can be easily evaluated by residues. After this we apply our arguments again, 
again, again. 

It is possible to wright the final expression (it is slightly long). Fortunately, 
this is not necessary. The only goal of our interest is 



lim(g^^+i(a;£) - ^^{a;e)) 

e— »0 

For instance, consider the summand obtained by the substitution 
Sa = a + Sa - -{p + q) + 1 + 2x 



(4.23) 



a 



-(p + q) + 1 + 2>f 



Then the integrand contains the factors 
1 



r(±i(s,-s,)) 



s^=a + e„-^{p + q) + l + 2x r(±i(£^-£„)) 
= a + £„ - |(p + g) + 1 + 2X- 



This factors tend to if £ ^ 0. Of course, it is necessary to check lacking of 
poles of the numerator in dangerous for us domain. 

Therefore a nonzero contribution to the limit (4.23) can be given only by 
the terms 



P r 

2 / Ha{a,s,s)dsi . . .dsa-ids„+i . . .dsp 



(4.24) 



Our expression is symmetric binder permutations of Sj and hence all sum- 
mands of (4.28) give the same contribution to the limit. Thus, deleting £ in 
(4.24) we obtain the formula 



27ri • 2p • 2« r 

nr(a-i + i)ip-i 



r(l(a-l(p + g) + l-si)) 
si- a + hip + l) - 1 - 2x 



r(i(a-i(p-hg) + l-hsi))x 



n r(i(a-i(p + ,) + l±.,)). n ^Mr-^^x 



r(±Sfc) 



n 



r(i(i + 5fc±gO)r(i(i-.sfc±sO) 

Ta{sk±si))rai-sk±si)) 



ds2 ■ ■ ■ dsp 



H=a-^{p+q) + l+2i<: 
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where a G Hi^+i fl O^. 

4.5. Calculations. Lemma 4.3.b is an obvious corollary of the last formula. 
Nevertheless we present some elements of the calculation, since it is essential 
for understanding Subsection 4.7. 



1) 



r(i(a+i(p + g)-l±Sfe)) 



r(i(si±sfe))r(|(-si±sfe)) 

^{-a + ^{p + q) - 1 ± Sk) - ft 
r(i(-a+i(p + g) + l±s,)) 



si=a-i(p+g)+2+2>t 



_ 2 



We observe that the factor Y{a; s) (see (2.2)) is canceled. This factor was the 
origin of singularities in our integral (2.1)-(2.5). 



2) r(i(i + si±sfc)) 



1, , , , =r(i(a-i(p + g) + l + 2x±Sfc)) 



We observe appearance of factor (4.8) which is very similar to the factor Y{a; s). 
Later it will be an origin of new singularities. 



3) 



r(i(a- i(p + g) + l + si)) 



si=a-i(p+(j) + l+2x 



r(si)r(-si) 

^ (-a+^(p + g)-2x-l) 
r(-a+i(p + g)-x) 

This gives formula (4.6)-(4.12), and completes the proof of Lemma 4.3. 

4.6. Analytic continuation through the line Rea < ^(p + g) — 1 — 2x. 

Lemma 4.3 gives the analytic continuation of to Os n IIj^+i. Evidentely the 
expression for the analytic continuation is analytic in the strip 



-2x- 1< a - i(p + g) + 1 < -2(x- 1) 



(4.25) 



and hence wo obtain the analytic continuation of to the whole strip (4.25). 

4.7. Proof of Theorem 2.2. The Plancherel formula (2.1)-(2.5) is correct 
if Re a > ^(p + 9) — 1- We want to construct the analytic continuation of its 
right part to the domain Rea < ^(p + g) — 1. Let us move a to the left side. 

Firstly we pass across the line a = ^{p+q) — l. Then we obtain the additional 
summand 5^°(a) := di{ct) — i?o(q:) given by formula (4.6)-(4.12) for k = 0. This 
is the summand of the Plancherel formula corresponding m = 1, mi=0. 

Let us compare the formula (4.6)-(4.12) for 5^°(a) and (2.1)-(2.5). First, we 
have in (4.6)-(4.12) additional factor (4.9). This factor has singularities but all 
these singularities are lying in the domain a > |(p + q) — 1. The factors (2.2) 
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and (4.8) are very similar {a is changed to a + 1). The factors (2.14)-(2.15) 
and (4.10)-(4.11) also are very similar. In fact (4.10)-(4.11) is the Gindikin- 
Karpclcvich density for 0{p — l,q — 1). 

Hence, we can construct the analytical continuation of i5°(a) in the same way 
as above. The first singularity of S^'^(a) on our way is the line Re a = 5 (p+q) — 2. 
After passing across the line we obtain one more summand ^^{a) corresponding 
m = 2, ui = U2 = 0. 

The line Re a = ^{p + q) — 3 contains singularities of the integral d{oi) and 
also singularities of 5^°" (a). Hence, we obtain two additional summands d^{a) 
and ^°^°{a) corresponding m = 1, Ui = 1 and m = 3, ui = ^2 = W3 = etc. 
etc. etc. 

Formally, we must give complete description of the inductive step but it 
literally repeats the arguments of Subsections 4.1-4.6. 

5. Positive definiteness of spherical functions. 

We obtained the expansion of !Ba(s) in spherical functions having the form 

'Ba{z)= ^s{z)diJ,ouris) (5.1) 

where the positive Dp-invariant measure ^,our{z) is described in Theorem 2.2. 
Our purpose is to prove positive definiteness of spherical functions $s (z) which 
are contained in the support of the measure fj^our- 

By the abstract Plancherel theorem, there exists the unique expansion 

'Ba{z)= ^s{z)dlJ,truth{s) (5.2) 

where Utruth is a positive Dp-invariant measure on C" supported by the space 

Gsph of positive definite spherical fimctions. 

Substitute z = to (5.2). Then S«(0) = 1, $^(0) = 1 and hence 

d^truth = 1 (5.3) 

We denote by supp ^itruth and supp ^our the supports of the measures ^itruth 
and fJjQuv 

5.1. Preliminary remarks on the supports of the measures. Consider 
the bounded polyhedron Q dW described in Theorem 1.14. Consider the tube 
Q C CP defined by the condition 

s€ Q iS Res € Q 
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. Then 

SUpp fitruth C Q; suppfiourCQ (5.4) 

(the first is corollary of Theorem 1.14, the second is corollary of Theorem 2.2). 
Denote by M U the union of the real and imaginary axises in C. Then 

supp flour C (M U «K) X • • • X (K U iR); supp fitmth C (M U «K) X • • • X (K U iM) 

(5.5) 

(the first is the corollary of Theorem 2.2 and the second is the Corollary of 
Lemma 1.13). 

5.2. Heat kernel. Let Ai, ... , Ap be Laplace operators (see |l^, Section 
2.5) on the symmetric space G/K. The operator Aj is some G-invariant partial 
differential operator of order 2j on G/K = Bp ^ with rational coefficients. The 
operator Ai is the usual Laplace-Beltrami operator on G/K (see 0, Section 
2.2.4). 

Spherical functions are joint eigenfunctions of the operators A^ (see JlTt , 
Section 4.2). We have equalities 

Aj<^>s{z) = aj{s)^s{z) 

where aj are some polynomials invariant with respect to the group Dp (see 
Subsection 1.15) consisting of permutations and changing of signs. If <f>s ^ <i>s', 
then aj(s) ^ o,j{s') for some j. 
In particular 

Ai$,(z) = (A + s2)$^(z) 

where A is a constant and 

2 2 I I 2 

s := Si H h Sp 

By conditions (5.4)-(5.5) the eigenvalues (A + s^) are real and they are uniformly 
bounded above on the supports of the measures flour, fJ-truth- 
Consider the Cauchy problem for the heat equation 

(|--Ai)F(z,t) -0; F{z,0)^f{z) 

on G/K. Let Rt{z,u) be the heat kernel. This means that the solution of the 
Cauchy problem for the heat equation is given by the formula 

F(z, r) = Arfiz) / i?,(z, u)f{u) dA(u) 

JG/K 

where A is the G-invariant measure on G/K. 
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Lemma 5.1. For each r > and N there exists a constant C{t,N) inde- 
pendent on z, u such that 

Rr{z,u) < C(r, iV)(l + dist(z,u))-" 

where dist(-,-) is the distance in G/K associated with Riemannian metric. 
Proof. Since the heat kernel is G-invariant, we can assume « = 0. Then 

R^{z, 0) = / exp{r(A + s^)}^s{z)ds 
J sex 

By integral formula for spherical functions (1.38) 

R^{z,0) = ( ( exp{T(A + s2)}^-^(^[k])(ikds 

Rapid decreasing of the last expression is more or less obvious. lEl 
Similar estimates are valid for partial derivatives of Rr{z,u) of any order. 
For spherical functions we have the equality 

Ar^,{z) = exp{r(A + s^)}^s{z) 

Lemma 5.2. Let iiour, ^J-truth be the same as above. Then 

a) Ar'Ba{z)^ exp{T{X + s^)}^s{z)diiouris) (5.6) 

Jg/k 

b) Ar'Ba{z)= [ exp{T{X + s^)}^s{z)dntruth{s) (5.7) 

JG/K 

Proof. We must prove a possibility to change the order of the integration. 
It is sufficient to show absolute convergence of the integrals 



Rr{z,u)^s{u) diJ,our{s)dX{u); / / Rr{z,u)^s{u)dfj,truthis)dX{u) 

G/K JCP JG/K JCP 

(5.8) 

a) For the first integral (5.8) we use estimate |$s(?i)| ^ <1>rcs(w) (sec (1.39)). 
The measure Hour is supported by a finite family of planes P^^-* having the form 

Pj-. Resi=9['\...,ReSm = 0ii^ 
On each plane Pj the integrand is dominated by some expression 

J2Rr{z,u)^„.{u) 

where aj are real vectors. The last expression doesn't depend on s. It remains to 
notice that the heat kernel rapidly decrease in u for fixed z, spherical functions 
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^a-j (w) are bounded (see Theorem 1.14), and the density of Hour (s) exponentially 
decreases if |s| oo. 

b) For the second integral we use the inequality |$s(-2)| ^ 1 and this com- 
pletes the proof (see (5.3)). lEl 

Lemma 5.3. For each polynomial r(xi, . . . , Xp) we have 

r(Ai, . . . , Ap)Ar'Ba{z) = / r(ai(s), . . . , ap{s)) exp{T(A + s^)}$s(z) d^Xour{s) 

Jg/k 

r(Ai, . . . , \)Ar'Ba{z) = I r(ai(s), . . . , ap{s)) exp{T(A + s^)}^s{z) dntruth{s) 

JG/K 

Proof. It is sufficient to prove that all partial derivatives by z of integrals 
(5.6), (5.7) absolutely converges. It is obvious by the following reasons. 

1. The integrand rapidly decreases in the variable s. 

2. For a given z partial derivatives of the heat kernel by u rapidly decrease. 

3. Spherical functions are bounded. lEl 

5.3. Proof of positive definiteness. Consider a e supp/Xour- Consider 
the function 

p 

r]{s) = exp{T(A + s^)} ^(a,(s) - aj{a)f 
Let M be the maximum of 77 on (M U iM) x • • • x (R U iR). Then the function 

as) = M-n{s) 

satisfies conditions 

({a) = M; C(.s) <M if s ^ for all w e Dp 

Consider the sequence of functions 

a(s) = Ck{M - 77(5))'= exp{T(A + s^)} 

where Ck is determined by the condition / £,k{s) dvour = 1- Obviously the 
sequence ^fe(s) converges to distribution J2weDp ~ wa). 
The function ^k (s) is a polynomial expression 

^fe(s) = Pk{ai{s), ap(s),exp{r(A + s^)}) 

Consider the operator 

Sfe :=Pfc(Ai,...,Ap,A^) 

By Lemma 5.3 we have 

Ek'Baiz) = / ^k{s)^s{z)duour{s) (5.9) 
Jg/k 

Ek'Baiz) = / ^k{s)^si.z)di^truthi.s) (5.10) 
Jg/k 
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We have ^k{s) ^ 0. Hence, by Lemma 1.5d and (5.10) the function Ek'Ba{s) 
is positive definite. By (5.9) the sequence Sfc'Ba(s) converges to ^cr{z). Thus, 
^aiz) is a point-wise hmit of positive definite functions and hence it is positive 
definite. 

6. Other series 

6.1. Hermitizations. Below we present the hst of hermitizations (see 
Subsection 0.1)0 

Each Riemannian noncompact classical symmetric space G/K can be real- 
ized as a matrix ballQ. A matrix ball is a space of all matrices of a given size 
over K = M, C, H with norm < 1 satisfying (or not satisfying) some symmetry 
condition. The list of matrix balls is given in the Table 1. 

Table 1 



G/K 


IK 


size 


condition 


G/K 


l.GL(n,R)/0(n,R) 


R 


nxn 


z=z* 


Sp(2n,R)/U(ri) 


2.0(p,<7)/0(p)xO(g) 


R 


pxq 




U(p,g)/U(p)xU(g) 


3.Sp(2n,R)/U(n) 


C 


nxn 


z=z* 


Sp(2n,R)/U(n)xSp(2n,R)/U(n) 


4.GL(n,C)/U(n) 


C 


nxn 


Z — Z* 


U(2n)/U(n)xU(n) 


5.0(n,C)/0(n) 


R 


nxn 


z=-z^ 


SO*(2n)/U(n) 


6.Sp(2n,C)/Sp(n) 


H 


nxn 


Z — — Z* 


Sp(4n,R)/U(2n) 


7.U(p,<7)/U(p)xU(<7) 


H 


pxq 




[U(p,<})/U(p) xU(g)] x [U(p,5)/U(p) xU(9)] 


8.GL(n,H)/Sp(n) 


H 


nxn 


z—z'' 


SO*(2n)/U(n) 


9.Sp(p,g)/Sp(p)xSp(<,) 


C 


pxq 




U(2p,2g)/U(2p)xU(2g) 


10.SO*(2n)/U(n) 


C 


nxn 


t 

Z — Z 


SO* (2n)/U(n) xSO* (2n)/U(n) 



The last column contains the hermitization G/K of G/K. The embedding 
G/K — > G/K in all cases is obvious (we must omit nonholomorphic condition 
to a matrix 2;, after this we obtain an hermitian matrix ball). 

Remark. Some spaces of small dimension are present in the left column two 
times (for instance Lobachevskii plane 0(2, l)/0(2) x 0(1)). Two associated 
hermitizations are different. 

^''Classical part of the list is contained in H.JafTee paper . Olshanskii in [Q, ^\ 
ohserved that all cases (including exceptional cases) can be easily reduced to Nagano work 
p9|. The list 1-18 is in one-to-one correspondence with the list of c omp ressive semigroups of 
symmetric spaces and with the list of causal symmetric spaces, see |42[ , luL The list 1-10 is 
in one-to-one correspondence with the list of real classical categories, see p2[ , Addendum A. 

^■^This observation is present in [ po| 
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Table 2 contains hcrmitizations related to future tubes 

Table 2 



G/K GjK 

11. SO(2,n)/0(n)xO(2) [S0(2,n)/0(n) X 0(2)] X [S0(2,n)/0(n) X 0(2)] 

12. SO(l,p)xO(l,g) SO(2,p+g) 



Exceptional hermitizationsQ] |^ are given in the Table 3 

Table 3 



13. 


£111/30(10) xSO(2) 


[£111/80(10) xSO(2)] X [BIII/SO(10) xSO(2)] 


14. 


£;vii/£;iiixso(2) 


[£;vii/£;iiixso(2)]x[£;vii/£;iiixso(2)] 


15. 


FII/Spin(9) 


£111/30(10) xSO(2) 


16. 


Sp(2,2)/Sp(2)xSp(2) 


£111/30(10) x30(2) 


17. 


GL(4,H)/Sp(2) 


£VII/£IIIxSO(2) 


18. 


BIVXR/F4 


£VII/£IIIxSO(2) 



6.2. Kernel representations. A kernel representatioif^ p of G is a restric- 
tion of a highest weight representation p of G to G. The constructive description 
of the scalar- valued kernel representations of 0(p, q) given in Subsections 1.11- 
1.12 is valid for all series 1-10. 

Below we discuss only scalar-valued kernel-representations. 

6.3. Plancherel formula for large a for the series 1-10 was obtained in 

mn , ... .. ^ 

Consider for simplicity nonhermitian case or hermitian casa^j a = (3. In all 

E\l\ and EYV are real form of i?6i i?VII is a real forms of Ej, and _FII is a real form of 
F4, sec 

^^Here we observe one of the cases when a phenomenon existing for classical groups doesn't 
exist for all exceptional groups 

■^^This definition was proposed in [ p3[ 

■^^The case of hermitian spaces 3,7,10,11 was considered by Berezin for a = /3, see 
notations in Subsection 0.2 (the proof is published in |52|), For hermitian case a ^ the 
Plancherel formula independently on |E5| was obtained by Zhang |pa]. Future tube case is 
simple exercise. The hermitian rase s Tx 14 are covered by | |52) ; the case 15 is reduced to 
one of Gindikin (1964) integrals |1 1| . Probably only for the exceptional cases 16, 17, 18 the 
formula is not known. Hence, the possibility to obtain the solution in "general case" (i.e. 
16—18) is yet preserved. 

^^For hermitian case a ^ fi we have the formula 

P 1 1 
C(a)C(/3) n r(-(a ~ + Sfc))r(-(/3 - h2 + Sfc))dlJ(s) 

where d£! is the Shimeno measure | |4s| ] . 
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these cases the Plancherel formula has the form 



P r 1 

rr cosh-" = / r\ Ti-ia-h+skmis)^sit)ds (e.i) 

where m is the rank of G, £H(s) is the Gindikin-Karpelevich density, E{a) is a 
meromorphic factor and h is a constant. In fact, h is the last point of square 
integrability. This means that Jq^j^ \'Sa{z)\'^dz is finite for a > h and infinite 
for a = h. 

6.4. The analytic continuation of the Plancherel formula. Our 

arguments from Sections 4~5 don't depend on series and they are vaUd for all 
series 1-10. 

In fact considerations of Section 4 prove that the following formal procedure 
gives correct result. We fix m — 0, 1, . . . ,p :— rank G and the collection of 
numbers wi ^ M2 ^ . . . u„i such that a + 2um + (m — 1) dimK < h 

Let 



'2 



Sk — a + h — 2uk — k dim ] 



(6.2) 



Sj^ —a. — h^2uk-[-k dim DC 

Then the analytic continuation of (6.1) has the form 

TTcosh-"tfc= V E{a) ) ' / QZ''^{s)ds^+^...ds^ (6.3) 

In the cases K = C, H this formula can be considered as a final result in a closed 
form. 

In the case K = M (G = 0(p, g), Sp(2n,M), GL(n,M)) substitutions (6.3) are 
impossible without cancellations and in this case formula (6.3) gives algorithmic 
procedure of calculation of Plancherel measure. 

Remark. For the groups U(p, q) it is easy to obtain the Plancherel formula 
(see the explicit final expression in |33| ) using Berezin-Karpelevich formula for 
spherical functions (see , pOt) a nd Molev unitarizability results |^ ; partially 
this idea was also realized in |18[|. 

6.5. Kernel representations of compact groups. Hermitization pro- 
cedure also is valid for compact Riemannian symmetric spaces. To obtain the 
list of hermitizations we must replace the groups G, G in Tables 1-3 to their 
compact forms. For instance the table 1 transforms to the following table 1' 
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Table 1' 



G/K 


K size 


condition 


G/K 


l'.U(n)/0(n,R) 


M n X n 




Sp(n)/U(n) 


2'.0(p+9)/0(p)xO(g) 


R pxg 




U(p+9)/U(p)xU(?) 


3'.Sp(n)/U(n) 


C n X n 


z=z^ 


Sp(n)/U(n)xSp(n)/U(n) 


4'.U(n)xU(ri)/U(ri) 


C n X n 


Z — Z* 


U(2n)/U(n)xU(n) 


5'.0(n)xO(n)/0(n) 


M n X n 


z=-z* 


0(2n)/U(n) 


6'.Sp(n)xSp(ri)/Sp(ri) 


H rixn 


Z — — Z* 


Sp(2n)/U(2n) 


7'.U(p+9)/U(p)xU(g) 


H pxg 




[U(p+9)/U(p) xU(<7)] X [U(p+g)/U(p) xU(g)] 


8'.U(2n)/Sp(n) 


H nxn 


z=z' 


0(2n)/U(n) 


9'.Sp(p+q)/Sp(p)xSp(g) 


C pxg 




U(2p+2g)/U(2p)xU(2g) 


10'.O(2n)/U(n) 


C nxn 


z=z* 


0(2n)/U(n) X 0(2n) /U(n) 



The construction of the kernel representation of 0{p + q) given in Subsection 
1.28 can be Hterally translated to all series 1-10. For this purpose we must 
replace the space Matp_g(R) by the space Mat of all matrices over K (see the 
second column) having the size given in the third column and satisfying the 
condition in the forth column. The group G acts on Mat by fractional linear 
transformations^. 

Integrals evaluated in |Q easily give Plancherel formulas for all kernel- 
representations in the cases I'-IO'. 

The case of hermitian symmetric spaces was earlier considered by Zhang 

m- 

6.6. Our terminology, a) The term "hermitization". The variant G/K 
is a real form of G/K (see |^) contradicts to the generally accepted usage of 
the term "real form", since G/K is not the complexification Gc/Kc of G/K. 

b) The term "kernel representation" or "Berezin kernel representation". 
There is no common term for this object. One possible term is "Berezin trans- 
form. It is not suitable in our situation, since there is no "transform" in this 
paper. 

Another term is "canonical representation" . The term "canonical representa- 
tion" can be used in many other senses. For instance in and several succes- 
sive papers of Vershik, Gelfand and Graev this term was used for multiplicative 
integral and also for some infinite divisible representations. The complex of 

^"More details on these models of Riemannian compact symmetric spaces are contained in 
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phenomena related to infinite divisibility has his own interest (see, for instance 
Chapter 10) but it has small relation with kernel representations. 
There is also the term "Berezin quantization". The "quantization" in Berezin 
sence is an operation on the space of functions. In our picture this operation 
can be defined only for hermitian case. 
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